Managed Service Provider Gone Bad: One Scary Story
In a tale worthy of any Halloween horror show, I found one MSP's service so shoddy that "mal-service" provider is more apropos than managed service provider.
What do you call a managed service provider, or MSP, that does such a disservice, so blatantly, that it jeopardizes a customer's high-availability network goals? Malpractice comes to mind, so "mal-service" seems appropriate to me.
After spending a couple of days doing site surveys at multiple locations, I had some questions for the MSP of which I speak. The basic "what kind of network are you deploying?" led to a slew of "why?" questions. That the MSP failed to answer my questions in necessary detail was a sure warning that it was providing substandard service.
In this case, I'm certain the MSP didn't perform due diligence -- given what I learned on my site surveys, I can find no other reasonable explanation for the decisions it made. Space at the customer properties was limited, which the MSP would have known if it did its discovery. The physical size of the appliance shown in the MSP's network diagram prohibited its use in most locations simply because no room, rack, or other space was large enough to accommodate it. How did the MSP plan on fitting its beastly wares into the limited space? I got no answer on that one.
So next I asked about its plans for providing backup power for the gear. To my disbelief, the MSP said it never provides battery backup and that it expects the customer to do this. "It's in the contract," the MSP told me. Floored, I questioned how any MSP expecting to deliver high-availability service could do so without installing its own uninterruptible power supplies (UPS). Again, I received the answer: "It's in the contract."
I had to see this contract, so I obtained a copy and read it in its entirety -- and then again and again a third time. I read it yet again a few days later, and then challenged one of my team members to read it and find the requirement or disclosure that the customer is responsible for providing UPS (battery backup power). Neither she nor I found mention of such a requirement other than in a referenced document for customer site requirements. This document contained a peculiar reference to "UPS State requirements." Even in the context of the reference, assuming the UPS has anything to do with battery backup makes no sense. Given the customer was in the medical field, could the MSP be referring to the Digital Imaging and Communications in Medicine standard? Maybe, maybe not!
Moving past the UPS concern, I turned my attention back to the overall network design and then to security. I wanted to know why the MSP chose to place routers intended for large enterprises in a location that didn't have even hundreds of devices -- even if you count five or 10 devices per user. And I wanted to know about its VPN plans.
On the latter, I noted that the network diagram showed an appliance at the corporate site for setting up VPN connections to other sites. The appliances at the remote sites would connect and offer a basic firewall feature but nothing further -- no deep packet inspection or intrusion prevention services. The MSP argued that each of the smaller VPN gateways did have some protection. Next, I asked, "Where's the firewall for the corporate office?" There wasn't one -- not that the MSP could explain its rationale for not having one. (If you want a chuckle, read my earlier post, Where's the Firewall?)
Forgiving the errors and looking past the lack of a UPS at every site, I continued on the path of trying to understand the network and make it work. So I asked the MSP about its ability to provide a second appliance at the corporate site or even at another site to act as a backup to the VPN concentrator in the event of a hardware failure. The response was, "We can ship you another one overnight." Hopefully the equipment doesn't fail on a Friday and that the shipment reaches the customer sooner rather than later.
Not all MSPs are equal. But when one fails to perform minimal site surveys or show an inkling that it understands the meaning of high availability, but is comfortable bloating hardware and ignoring security and single points of failure, it is, in fact, deserving of the label "mal-service provider."
Do you have any of your own networking horror stories to share? Share them below.