SHARE



ABOUT THE AUTHOR


Gary Audin
Gary Audin is the President of Delphi, Inc. He has more than 40 years of computer, communications and security...
Read Full Bio >>
SHARE



Gary Audin | October 21, 2013 |

 
   

8 More Disaster Recovery Duds

8 More Disaster Recovery Duds The lesson: Test your backup systems, get management buy-in, and seek input from non-technical people to find the gaps in your plans.

The lesson: Test your backup systems, get management buy-in, and seek input from non-technical people to find the gaps in your plans.

Planning for business continuity is planning for interruptions to business operations. There will be disasters, foreseen and unforeseen. The unforeseen and neglected problems are what I want to discuss.

In my previous blog, Disaster Recovery Duds, I presented 8 disaster recovery duds that generated a lot of interest. So I have 8 more stories of disaster recovery planning and execution that did not go as planned.

When Did We Last Try This?--An enterprise installed a backup power generator. It was sized properly for the data center power requirements. The generator was tested and worked as planned. Everyone was very comfortable with the backup power arrangement.

Then it happened. The power was cut off. The generator started and provided power for about one minute, then stopped. It would not restart. It took quite a while to discover that the fuel filter was clogged with rust. The generator had not been exercised periodically (once a month for an hour would be best). The fuel tank had rusted and caused the filter to block the fuel flow.

Always, always exercise your backup facilities to ensure that they are working properly. Don't wait for the disaster to learn how well the backup works.

The UPS Will Fit into the Closet--As network closets become a power source with PoE, then the UPS battery backup also moves to the network closet. An enterprise budgeted properly for the closet UPS, but the network closet did not have enough floor space to allow the installation of the UPS batteries. So one of the IT staffed decided to locate the batteries vertically along one wall. The facilities people found out and immediately stopped the UPS installation. The UPS, when installed vertically, exceeded the weight limitations for the floor. Before the UPS could be installed, the floors below the closet had to be reinforced to hold the weight of the batteries. This worked; however the extra cost for the reinforcement caused the UPS budget to be far exceeded.

The Vent is Where?--A diesel generator was installed successfully as a backup power source. As the generator was being tested, someone walked into the closed shipping and receiving garage. That person discovered that the generator was vented into the garage, not outside. The exhaust design had to be changed, which increased the budget.

The Data Center was Safe--A new data center was constructed in an existing office building. The planners anticipated potential building closure due to scenarios such as bad weather, protesters, and planned for employees staying on site for days; they planned for backup power, but not a thought about the building. The data center was the hub for 30 remote data centers and 14 networks. The data center was above the building loading dock. A single critical pillar on the corner of the loading dock was just below the data center. If a truck damaged or collapsed this pillar, then the data center would be destroyed.

The Data Center Backup Worked Fine--An information services enterprise planned and executed a successful backup power arrangement. It worked as expected. However, it was tested when the primary building power was still on. The backup power provided excellent isolation from the utility company's power which had many short power fluctuations.

Finally, the primary power was lost one evening.

What the backup power designers did not consider was the power to the security scanners for entry into the building. No one could enter the building because the scanners were inoperable. Those who wanted to enter had to call on their cell phones to those inside to get someone to open the door for them to enter.

Remember, there are always other power vulnerabilities. Test the backup power under real conditions to locate the missed vulnerabilities.

Fueling the Generator--A lesson learned during Hurricane Katrina was that roads could be closed for weeks. Diesel generators worked for several days. When the diesel fuel was exhausted, it was learned that the refueling trucks could not get to the generator locations. The assumption that they could be refueled before the generators stopped running was incorrect.

In response, a school system planned their backup power generators so they could run on natural gas through pipelines. It was less likely that the pipelines would be affected by a storm and thus could run indefinitely.

The Disaster Recovery Staff--One of my early projects was the automation of a central bank. It was very important that the operations should continue during a failure even if they were to be manually operated.

So we designed a manual backup system with equipment and staff to support it. We never needed the manual system because the design of the automated system worked under all conditions.

One day I decided that we needed to test the manual backup system. It did not work. And in the meantime, the departments that were responsible for the manual backup staff had gained so much confidence in the automated system that the backup staff was reassigned without notifying the IT department. The equipment was there but not the people. If you create any backup that depends on extra or reassigned staff, make sure they really exist and are prepared for the failure.

The Emergency Host--An organization decided they needed to update their emergency response procedures. They proposed that every building on their campus have an emergency host that would stand outside the building to direct emergency personnel when they arrived. Sounds good until you consider explosions, chemical leaks, earthquakes, and terrorism. The plan assumed that the host would only be needed during office hours. The plan was inexpensive to execute but not really workable. Keep in mind, people may not behave as planned during an emergency unless they are well trained and the emergency procedures are frequently exercised.

There never can be a perfect plan that covers all possibilities. When the plan is written, give copies to as many management level personnel as possible, especially those most affected by a disaster. Non-technical feedback can be useful for locating holes in the plan.





COMMENTS



August 16, 2017

Contact centers have long been at the leading edge of innovation in communications technology, given their promise of measurable ROI and the continual need to optimize customer interactions and sta

July 12, 2017

Enterprises have been migrating Unified Communications & Collaboration applications to datacenters - private clouds - for the past few years. With this move comes the opportunity to leverage da

May 31, 2017

In the days of old, people in suits used to meet at a boardroom table to update each other on their work. Including a remote colleague meant setting a conference phone on the table for in-person pa

August 16, 2017
World Vision U.S. is finding lots of goodness in RingCentral's cloud communications service, but as Randy Boyd, infrastructure architect at the global humanitarian nonprofit, tells us, he and his team....
August 11, 2017
Alicia Gee, director of unified communications at Sutter Physician Services, oversees the technical team supporting a 1,000-agent contact center running on Genesys PureConnect. She catches us up on th....
August 4, 2017
Andrew Prokop, communications evangelist with Arrow Systems Integration, has lately been working on integrating enterprise communications into Internet of Things ecosystems. He shares examples and off....
July 27, 2017
Industry watcher Elka Popova, a Frost & Sullivan program director, shares her perspective on this acquisition, discussing Mitel's market positioning, why the move makes sense, and more.
July 14, 2017
Lantre Barr, founder and CEO of Blacc Spot Media, urges any enterprise that's been on the fence about integrating real-time communications into business workflows to jump off and get started. Tune and....
June 28, 2017
Communications expert Tsahi Levent-Levi, author of the popular BlogGeek.me blog, keeps a running tally and comprehensive overview of communications platform-as-a-service offerings in his "Choosing a W....
June 9, 2017
If you think telecom expense management applies to nothing more than business phone lines, think again. Hyoun Park, founder and principal investigator with technology advisory Amalgam Insights, tells ....
June 2, 2017
Enterprises strategizing on mobility today, including for internal collaboration, don't have the luxury of learning as they go. Tony Rizzo, enterprise mobility specialist with Blue Hill Research, expl....
May 24, 2017
Mark Winther, head of IDC's global telecom consulting practice, gives us his take on how CPaaS providers evolve beyond the basic building blocks and address maturing enterprise needs.
May 18, 2017
Diane Myers, senior research director at IHS Markit, walks us through her 2017 UC-as-a-service report... and shares what might be to come in 2018.
April 28, 2017
Change isn't easy, but it is necessary. Tune in for advice and perspective from Zeus Kerravala, co-author of a "Digital Transformation for Dummies" special edition.
April 20, 2017
Robin Gareiss, president of Nemertes Research, shares insight gleaned from the firm's 12th annual UCC Total Cost of Operations study.
March 23, 2017
Tim Banting, of Current Analysis, gives us a peek into what the next three years will bring in advance of his Enterprise Connect session exploring the question: Will there be a new model for enterpris....
March 15, 2017
Andrew Prokop, communications evangelist with Arrow Systems Integration, discusses the evolving role of the all-important session border controller.
March 9, 2017
Organizer Alan Quayle gives us the lowdown on programmable communications and all you need to know about participating in this pre-Enterprise Connect hackathon.
March 3, 2017
From protecting against new vulnerabilities to keeping security assessments up to date, security consultant Mark Collier shares tips on how best to protect your UC systems.
February 24, 2017
UC analyst Blair Pleasant sorts through the myriad cloud architectural models underlying UCaaS and CCaaS offerings, and explains why knowing the differences matter.
February 17, 2017
From the most basics of basics to the hidden gotchas, UC consultant Melissa Swartz helps demystify the complex world of SIP trunking.
February 7, 2017
UC&C consultant Kevin Kieller, a partner at enableUC, shares pointers for making the right architectural choices for your Skype for Business deployment.
February 1, 2017
Elka Popova, a Frost & Sullivan program director, shares a status report on the UCaaS market today and offers her perspective on what large enterprises need before committing to UC in the cloud.
January 26, 2017
Andrew Davis, co-founder of Wainhouse Research and chair of the Video track at Enterprise Connect 2017, sorts through the myriad cloud video service options and shares how to tell if your choice is en....
January 23, 2017
Sheila McGee-Smith, Contact Center/Customer Experience track chair for Enterprise Connect 2017, tells us what we need to know about the role cloud software is playing in contact centers today.