Surviving Irma: Lessons Learned, Personal & ProfessionalSurviving Irma: Lessons Learned, Personal & Professional
We count our blessings, learn from our mistakes... and get ready for next time.
September 15, 2017
I'm reporting in from south Florida, thankful to be able to be in the position to share some lessons learned. But first, a word about about the "psychosis."
Everyone north of Florida seems to be terrified of hurricanes. Last year, Matthew (careful what you say) skimmed by my community and he was no laughing matter. Had he changed course a wee bit then, yes, our community would have been devastated. Of course Floridians respect hurricanes, but somehow the national news media managed to sensationalize coverage as they depicted the "horrors" for viewers in the rest of the country (as I recall from high school, this is known as "yellow journalism"). Someone showed me a media clip of a broadcast concerning hurricane Matthew in which a well-known national reporter told Floridians to get their children out or they would die.
My heart goes out to residents of the Florida Keys and in the many other communities hit by hurricane Irma.
I spent Saturday, Sept. 2, installing fiber on a campus, not knowing about the threat of Irma. I got the news when I arrived home in the evening, and the very first mistake I made was not immediately heading out to buy LP gas, water, and some provisions. LP gas and water were sold out by early the next afternoon, and provisions were quickly leaving the store shelves. Gas prices hiked, not due to Irma but as an aftermath of hurricane Harvey's strike on Texas and disruption to the supply line. Fortunately, we were able to get a spare tank filled by a gas company minutes before it closed on Friday, Sept. 8. We then put up our aluminum shutters to cover and protect our window and door openings -- a big chore (when we did this last year the humidity was so high, I really felt the pressure of my age).
As I noted last year in a post-Matthew piece, MyRadar is an excellent app to use during a weather event. It broadcasts warnings on things like tornado and floods, which did impact my community during Irma. This year I also used another app, called Barometer & Altimeter, as well as a hygrometer, which measures water vapor in the atmosphere. (During Matthew, I kept telling my family that the barometric pressure wasn't "right," meaning it never dropped enough to rate as "hurricane" -- or even within a range of what would have been normal for a hurricane directly hitting our community.)
Floridians have a plethora of information and resources available to them when it comes to natural disasters. While I had overlooked these, I do know that the annual Governor's Hurricane Conference for first responders and disaster relief agencies yields positive best practices. I saw this in the communications delivered via local news, the broadcasting of advanced warnings through email and radio, and other updates from reliable sources of information before, during, and after the storm.
Source: Wikipedia
Now to the business side of things -- and if you gleaned anything from the personal story, I hope it's to act in a timely manner because human latency can be catastrophic.
Click to the next page for lessons learned
Continued from Page 1
Here are takeways to remember:
Radio systems are imperative to many organizations. Expect to be without them unless you have pre-established some centralized charging stations that have uninterrupted power.
You don't want your generator running on fumes. People expect a generator to run forever, but that presumes that the business has an endless supply of fuel for it. Another issue to consider for diesel generators is that runtime calculations cannot include the fuel tank's full capacity. That's because when generators suck too much air, they fail -- and restarting a diesel generator when the fuel injection system has air in it takes some work. If you have a 500-gallon tank, then don't assume you can run 480 gallons before refilling. If you do, you'll chance stalling (sucking air into the fuel injection system) and then creating new problems for yourself if you do happen to be able to refuel the generator.
Another suggestion is to limit the generator's use to essential items in order to maximize uptime. To calculate runtime and maximize efficiency, determine your power load, and whatever that number is add a buffer of 15% to it because it's never right. Some IT gear is very inefficient.
Power for IDFs and MDFs or closets should be aligned. Having 15 minutes of UPS power when you have 30 hours of diesel generator power is just wrong. What can you get done in 15 minutes? Secondly, refueling takes longer than that -- and again you run risks of introducing air into the system and causing a shutdown that can take hours to repair. The IDFs and MDF, often become mismatched in runtime, and when the MDF has one hour of battery and the IDFs have two, you can guess that nothing is going to work from the MDF.
Establish an order for shutting down equipment should fuel be low, and for bringing systems back online once power is restored. Does it make a difference? Most likely yes, as servers are interdependent as are some applications. Getting the order right will minimize your planned outage or time to restore, while getting it wrong... well it's more latency.
Account for business processes such as payroll. One national fast food chain lost its payroll support center in Florida. Will employees get paid on time? Unfortunately, many business processes are disrupted during storms like Irma. What about the engineering management system that seems to be missing something and you can't get a chiller back online or some other component mission critical to the building? Lack of documentation is a problem. Any IT person or any building engineer or electrician, whatever role is applicable, should be able to read documentation and carry out a shutdown or recovery effort.
People must be able to perceive the risk or danger before they act, because in many examples they do not act immediately. When they do act immediately, as in this case, highways can be congested, gas lines long, pumps go dry, and people become distraught. Some might see this as a punishment of sorts for doing the timely thing, but it's a part of a bigger process.
Many national companies have acted in good faith. Comcast provided free Wi-Fi to anyone in Florida. Geico extended time to pay insurance renewals, as did Chase for credit card bills and mortgages. And, AT&T didn't charge for texts to and data for those in peril.
This storm taught me a lot, things I can apply to the inevitable next time. And when that next time arrives, I'll be in learning mode again.
We are thankful we came through Irma safely -- please make no mistake about that (having to take several cold showers is not a bad thing). So we count our blessings and learn from our mistakes.
Follow Matt Brunk on Twitter!
@telecomworx
