Bad Voice: Untangling a Network Mess
Helping one company figure out why voice quality was so flaky
As a recent client engagement reminded me, site assessments are imperative for revealing trouble that could create roadblocks but will often require considerable investment of time and money if no clear network architecture or roadmap is in place.
In such situations, the first step is to obtain bills and contracts for carrier services and equipment. Having this information will help you build an accurate picture of what's in place when folks onsite may or may not be able to explain what they have and why. A good second step is the basic one of taking inventory. Then comes mapping connectivity between sites and services. These later steps require onsite presence and, as I've written previously, the best practice is to verify the services at the demarc.
For the situation I recently encountered, my theory is that vendors capitulated to work-arounds in attempts to fix network connections that weren't meant to work. Mix in a messy M&A, and inherited resources, and the problem grew. Here's what I discovered:
- Bonded T1s contracted with a carrier now gone from the scene led to an on-premises IP-PBX sending traffic out a voice path but not receiving traffic from numerous branch offices across a different connection
- Sparse programming remnants on premises hardware (switches, router, firewall) show evidence of attempts to set up virtual LANs
- A session border controller sitting between the spans and the firewall lacked adequate licensing for "simultaneous calls"
- Misconfigured firewall failover and load balancing, with a surplus of bandwidth compounding the problems (even though that bandwidth was probably meant to be a solution)
- Numerous off-the-shelf switches sat between hardware, apparently implemented as "Band-Aids"
Interestingly, I also found "MPLS" tagged on the smart jacks and the gears' programmed ports, as well as within a contractual reference. And that leaves me wondering about the softswitch destination of the SIP traffic to a carrier location at a data center. While no MPLS network was actually in place, the idea of separating voice and data traffic came into reality because the involved parties (IT, IT subcontractor, PBX vendor, PBX subcontractor, and three carriers) abandoned attempts at properly routing traffic through the network . They simply couldn't figure out a proper configuration.
These discoveries stemmed from complaints such as "voice never works right" for the satellite offices and that users experience "cut-offs and strange noises." The other complaints were the slowness of applications and, for a firm engaged in processes that must be turned around with a sense of urgency and oversight (compliance), the network in hand just didn't lend itself to repair.
The deeper you have to go on an assessment is usually indicative of more problems -- and that was the case here. Lack of consistency and shoddy work in cabling, power, housekeeping, programming, and installation all circled back to core problems and vendors that weren't qualified to do the work. With no clear plan or a network design laid out, this company was not able to carry out its core mission without daily disruption.