One of my customers is a $20M contracting company, and their facility is like most others. Their structured wiring was improved about two years ago when we did our first site survey. However, despite previous recommendations, their infrastructure was not improved.
The network was reported down in the morning. A dead firewall, Comcast router and Verizon T1 router meant that the network was hard down. Fortunately, the voice service remained live with only a few dead circuits; but the company isn't positioned to operate on voice only mode.
After four hours of troubleshooting, the day ended with both Comcast and Verizon unable to make a site visit. The second day advanced with the HVAC tech reporting that the building lost a phase on their 3-phase power, and the needed air conditioning couldn't be restored. After Comcast and Verizon visited the site, the Comcast tech failed to properly configure the cable modem and the Verizon tech failed to replace the router, saying it wasn't covered under their "enterprise contract." We met again onsite and configured the cable modem and then started troubleshooting new issues involving a $20K plotter that burned up.
The lightning storm damaged the coaxial cabling, POTS and T1 services, rendering equipment useless. The loss of a phase was due to the damaged transformer that sustained the lightning hit.
The new interest in whole-panel protection was met with no resistance when the customer said, "Just tell us what to do so we can avoid the loss of our time again." I've written about power and protection previously, and each summer I find new opportunities to report on the substantial losses that businesses continue to incur when they fail to improve and protect their infrastructures.
Our recommendations didn't stop with whole-panel protection, since we reviewed with the customer the discrepancies we noted on the site review. A new DVR security system was recently installed, sharing the network and PBX power--and having two UPSs on one circuit is never a good idea. The power was also overloaded with the sump pump, which we found connected to the same circuit serving the DVR, network and PBX.
Next week as we return to perform more repairs, I can't help but think that "Here we go again" applies to us as much as it does to the customers that hear again from us what needs to be done to protect their investments and to keep their networks up.
Follow Matt Brunk on Twitter and Google+!
@telecomworx
Matt Brunk on Google+
