Terry Slattery
Terry Slattery, is a senior network engineer with decades of experience in the internetworking industry. Prior to joining Chesapeake NetCraftsmen as...
Read Full Bio >>

Terry Slattery | November 26, 2013 |


Troubleshooting VoIP Packet Loss

Troubleshooting VoIP Packet Loss A straightforward approach to troubleshooting VoIP that can be applied to video as well.

A straightforward approach to troubleshooting VoIP that can be applied to video as well.

VoIP is particularly sensitive to packet loss, and determining the cause of the packet loss can be difficult. Each vendor has different tools that may be used to help with the diagnosis. I'm going to cover some of the causes of packet loss and describe some of the tools that exist. Of course, contacting the vendor will provide you with more information about any new tools or troubleshooting methodologies that are specific to that vendor.

Packet loss in VoIP will typically have a slowly degrading impact on speech communications. The human ear is very good at handling the short gaps that are typical of packet loss. So it may take a significant amount of packet loss for the user community to be annoyed enough to report it.

It is best to use automated management systems to collect the packet loss data from the VoIP system, allowing you to generate reports that you can use to determine the scope of any loss (see below). However, note that fax communications isn't tolerant of any packet loss, so don't try to run fax machines over VoIP. Finally, the principles below can be applied to video troubleshooting, since many of the same mechanisms are in use for interactive video.

Determine the scope of the problem
First, it is important to gain some understanding of the scope of the problem. Is the packet loss restricted to a subset of all VoIP endpoints, or is it occurring across all endpoints?

It is useful to think of groups of endpoints. A simple group is the endpoints at a remote site. A more complex group is the set of endpoints that communicate with external peers, such as calls out to the PSTN. There might even be problems with a particular gateway to the PSTN, indicated by problems with only those calls that traverse that gateway. If the problems seem to appear across all endpoints, perhaps they have a systemic origin, such as a missing or incorrect QoS configuration.

Don't forget about media gateways and MCUs. If the enterprise's conference calls are more prone to voice dropouts, then there may be a problem related to one of the MCUs or the network around it. In fact, there may be multiple small problems and configuration errors that interact to create significant packet loss. For example, a firewall may be configured to drop out-of-order fragments, while at the same time a part of the network is configured to use per-packet load balancing, which may prove to be a source of out-of-order packets.

Second, determine if the problem is time-dependent. Can the problem be correlated with other network or VoIP system events? If it is happening all the time, then you can eliminate many causes that would only occur at a specific time of day, such as packet loss due to congestion during busy times of the day.

The combination of scope and time may provide clues to the origin of the packet loss. At a minimum, the data should allow you to eliminate a lot of potential problems, making it easier to eventually identify the core problem. Keep in mind that multiple minor problems may be interacting to cause the packet loss.

Sources and types of packet loss
There are two types of packet loss in a VoIP system: Receive Packet Loss, and Receive Packet Discard. Receive Packet Loss is where a packet is never delivered to the receiving system, while Receive Packet Discard is where a packet is received at a time when it is not useable for generating audio playback.

VoIP-receiving endpoints typically record both Packet Loss and Packet Discard counters, allowing you to determine which type of problem you are troubleshooting. Cisco phones include these stats in either the manual user interface on the phone set, or through its web interface. Avaya phones report these stats via RTCP, which at one time required a separate RTCP receiver to record the stats. As VoIP control software is revised, you may find these stats available within the VoIP controller.

Receive Packet Loss means that the packet is dropped somewhere in the network. In this case, you'll be looking for problems that cause packets to be discarded within the network.

1. It could be a bad link that is causing packet errors, which would affect any connections via that network link. This problem may not vary by time of day or load. A WAN connection may exhibit continuous errors while a duplex mismatch could cause errors that are load-dependent.

2. Network congestion without QoS could create packet loss in extreme cases where router or switch buffers overflow.

3. A transient network problem like a flapping link can cause convergence problems in routing protocols or in the Spanning Tree protocol used in switch-based networks. Packets will get dropped if a valid alternate path is not immediately available.

4. In rare cases involving MCUs, packet loss could be due to an overloaded software-based MCU or a problem with a Digital Signal Processor (DSP) in a hardware-based MCU.

Receive Packet Discard occurs when the packet arrives at the receiver at a time when the packet cannot be used for playback. It is more typical for a packet to arrive late than it is for a packet to arrive too early to store into the playback buffer. When a packet arrives, but it can't be played back, it is discarded and the Discard counter is incremented.

1. A network without QoS or with an incorrect QoS configuration may cause high jitter during periods of congestion, when the routers and switches need to buffer packets. High jitter typically causes packets to arrive too late to be played back.

2. Out-of-order packets are also typically discarded by VoIP endpoints. Packet ordering can be affected by per-packet load balancing over parallel paths. It can also happen when routing changes cause an alternate path to be used, though this is typically a transient event rather that an ongoing problem. Flapping links, however, can cause out-of-order problems on a more continuous basis.

Because some of the above problems are transient, look for Loss or Discard statistics that are regularly increasing and have relatively high values. For example, a Discard counter that's increasing by thousands per day would be reason for action, while an increase of less than 1,000 per day would only add it to my watch list.

Detecting packet loss
Each vendor implements a different mechanism for reporting packet loss. Cisco phones report their statistics back to the call controller, where the values are used to calculate voice quality. The phone keyboard can be used to examine the packet loss counters, or a remote web interface can be used to examine the same counters. The Cisco Unified Call Manager (the call controller) uses packet loss statistics to calculate the overall voice MOS score, but doesn't make the specific values available. So the best way to find problems in a Cisco VoIP implementation is to look at reports for calls or endpoints with consistently low MOS scores.

Avaya uses its VoIP Monitoring Manager to receive the RTCP stream from VoIP endpoints. The RTCP data includes packet loss statistics, so it is easy to generate reports on endpoints with the highest packet loss.

Other vendors have similar packet loss reporting mechanisms, either within their own products, or provided by third-party VoIP analysis and reporting systems. Of course, end user reports of dropouts in calls is an important factor, especially since a significant amount of packet loss will have to exist before it really begins to affect voice quality.

I think it is imperative that anyone running a VoIP system should have the tools installed to monitor all call quality parameters, particularly packet loss/discard statistics. Examining the call quality reports regularly, and proactively identifying packet loss and rectifying the causes, will keep your VoIP network running smoothly.


Enterprise Connect Orlando 2018
March 12-15 | Orlando, FL

Connect with the Entire Enterprise Communications & Collaboration Ecosystem

Stay Up-to-Date: Hear industry visionaries in Keynotes and General Sessions delivering the latest insight on UC, mobility, collaboration and cloud

Grow Your Network: Connect with the largest gathering of enterprise IT and business leaders and influencers

Learn From Industry Leaders: Attend a full range of Conference Sessions, Free Programs and Special Events

Evaluate All Your Options: Engage with 190+ of the leading equipment, software and service providers

Have Fun! Mingle with sponsors, exhibitors, attendees, guest speakers and industry players during evening receptions

Register now with code NOJITTEREB to save $200 Off Advance Rates or get a FREE Expo Pass!

March 7, 2018

Video collaboration is experiencing significant change and innovation-how can your enterprise take advantage? In this webinar, leading industry analyst Ira Weinstein will present detailed analysis

February 21, 2018

Business agility has become the strongest driver for enterprises to begin migrating their communications to the cloud-and its a benefit that enterprises are already realizing. To gain this benefit

February 7, 2018

Enterprises are starting to grasp the critical importance of security and compliance in their team collaboration deployments. And once the risks are mitigated, your enterprise can integrate these n

March 12, 2018
An effective E-911 implementation doesn't just happen; it takes a solid strategy. Tune in for tips from IT expert Irwin Lazar, of Nemertes Research.
March 9, 2018
IT consultant Steve Leaden lays out the whys and how-tos of getting the green light for your convergence strategy.
March 7, 2018
In advance of his speech tech tutorial at EC18, communications analyst Jon Arnold explores what voice means in a post-PBX world.
February 28, 2018
Voice engagement isn't about a simple phone call any longer, but rather a conversational experience that crosses from one channel to the next, as Daniel Hong, a VP and research director with Forrester....
February 16, 2018
What trends and technologies should you be up on for your contact center? Sheila McGee-Smith, Contact Center & Customer Experience track chair for Enterprise Connect 2018, gives us the lowdown.
February 9, 2018
Melanie Turek, VP of connected work research at Frost & Sullivan, walks us through key components -- and sticking points -- of customer-oriented digital transformation projects.
February 2, 2018
UC consultant Marty Parker has crunched lots of numbers evaluating UC options; tune in for what he's learned and tips for your own analysis.
January 26, 2018
Don't miss out on the fun! Organizer Alan Quayle shares details of his pre-Enterprise Connect hackathon, TADHack-mini '18, showcasing programmable communications.
December 20, 2017
Kevin Kieller, partner with enableUC, provides advice on how to move forward with your Skype for Business and Teams deployments.
December 20, 2017
Zeus Kerravala, principal analyst with ZK Research, shares his perspective on artificial intelligence and the future of team collaboration.
December 20, 2017
Delanda Coleman, Microsoft senior marketing manager, explains the Teams vision and shares use case examples.
November 30, 2017
With a ruling on the FCC's proposed order to dismantle the Open Internet Order expected this month, communications technology attorney Martha Buyer walks us through what's at stake.
October 23, 2017
Wondering which Office 365 collaboration tool to use when? Get quick pointers from CBT Nuggets instructor Simona Millham.
September 22, 2017
In this podcast, we explore the future of work with Robert Brown, AVP of the Cognizant Center for the Future of Work, who helps us answer the question, "What do we do when machines do everything?"
September 8, 2017
Greg Collins, a technology analyst and strategist with Exact Ventures, delivers a status report on 5G implementation plans and tells enterprises why they shouldn't wait to move ahead on potential use ....
August 25, 2017
Find out what business considerations are driving the SIP trunking market today, and learn a bit about how satisfied enterprises are with their providers. We talk with John Malone, president of The Ea....
August 16, 2017
World Vision U.S. is finding lots of goodness in RingCentral's cloud communications service, but as Randy Boyd, infrastructure architect at the global humanitarian nonprofit, tells us, he and his team....
August 11, 2017
Alicia Gee, director of unified communications at Sutter Physician Services, oversees the technical team supporting a 1,000-agent contact center running on Genesys PureConnect. She catches us up on th....
August 4, 2017
Andrew Prokop, communications evangelist with Arrow Systems Integration, has lately been working on integrating enterprise communications into Internet of Things ecosystems. He shares examples and off....
July 27, 2017
Industry watcher Elka Popova, a Frost & Sullivan program director, shares her perspective on this acquisition, discussing Mitel's market positioning, why the move makes sense, and more.
July 14, 2017
Lantre Barr, founder and CEO of Blacc Spot Media, urges any enterprise that's been on the fence about integrating real-time communications into business workflows to jump off and get started. Tune and....
June 28, 2017
Communications expert Tsahi Levent-Levi, author of the popular blog, keeps a running tally and comprehensive overview of communications platform-as-a-service offerings in his "Choosing a W....
June 9, 2017
If you think telecom expense management applies to nothing more than business phone lines, think again. Hyoun Park, founder and principal investigator with technology advisory Amalgam Insights, tells ....
June 2, 2017
Enterprises strategizing on mobility today, including for internal collaboration, don't have the luxury of learning as they go. Tony Rizzo, enterprise mobility specialist with Blue Hill Research, expl....
May 24, 2017
Mark Winther, head of IDC's global telecom consulting practice, gives us his take on how CPaaS providers evolve beyond the basic building blocks and address maturing enterprise needs.
May 18, 2017
Diane Myers, senior research director at IHS Markit, walks us through her 2017 UC-as-a-service report... and shares what might be to come in 2018.
April 28, 2017
Change isn't easy, but it is necessary. Tune in for advice and perspective from Zeus Kerravala, co-author of a "Digital Transformation for Dummies" special edition.
April 20, 2017
Robin Gareiss, president of Nemertes Research, shares insight gleaned from the firm's 12th annual UCC Total Cost of Operations study.
March 23, 2017
Tim Banting, of Current Analysis, gives us a peek into what the next three years will bring in advance of his Enterprise Connect session exploring the question: Will there be a new model for enterpris....
March 15, 2017
Andrew Prokop, communications evangelist with Arrow Systems Integration, discusses the evolving role of the all-important session border controller.
March 9, 2017
Organizer Alan Quayle gives us the lowdown on programmable communications and all you need to know about participating in this pre-Enterprise Connect hackathon.
March 3, 2017
From protecting against new vulnerabilities to keeping security assessments up to date, security consultant Mark Collier shares tips on how best to protect your UC systems.
February 24, 2017
UC analyst Blair Pleasant sorts through the myriad cloud architectural models underlying UCaaS and CCaaS offerings, and explains why knowing the differences matter.
February 17, 2017
From the most basics of basics to the hidden gotchas, UC consultant Melissa Swartz helps demystify the complex world of SIP trunking.
February 7, 2017
UC&C consultant Kevin Kieller, a partner at enableUC, shares pointers for making the right architectural choices for your Skype for Business deployment.
February 1, 2017
Elka Popova, a Frost & Sullivan program director, shares a status report on the UCaaS market today and offers her perspective on what large enterprises need before committing to UC in the cloud.
January 26, 2017
Andrew Davis, co-founder of Wainhouse Research and chair of the Video track at Enterprise Connect 2017, sorts through the myriad cloud video service options and shares how to tell if your choice is en....
January 23, 2017
Sheila McGee-Smith, Contact Center/Customer Experience track chair for Enterprise Connect 2017, tells us what we need to know about the role cloud software is playing in contact centers today.