The Killer Duplex Mismatch
This technical anomaly haunts our networks and refuses to go away. It has been the bane of many a network administrator and voice or video vendor when real-time traffic is first introduced to the network.
One of the most common network problems for voice and video conferencing is the duplex mismatch. This technical anomaly haunts our networks and refuses to go away. It has been the bane of many a network administrator and voice or video vendor when real-time traffic is first introduced to the network.
A duplex mismatch occurs when two devices connected by Ethernet do not properly negotiate their connection. Ethernet has the option of running at different speeds (10, 100, or 1 Gbps) and has the option of running half duplex or full duplex. Choosing a mode of operation occurs when the cable is first connected or when an endpoint is first powered up. It is determined by a protocol negotiation between the two endpoints, which in theory should find the highest speed, and should choose full duplex if it is available and half duplex if full duplex is not available. In some cases that negotiation fails, and one end decides to run full duplex while the other end decides to run half duplex. Because the two endpoints are not running a common protocol, packet loss occurs.
This duplex mismatch often occurs between an endpoint and the access switch of the network, although mismatches can also occur between network devices such as switches or between switches and routers. Very often these duplex mismatches go undetected until a voice or video system is installed.
A duplex mismatch will cause constant packet loss. Because TCP has an excellent packet recovery mechanism, data traffic often crosses this duplex mismatch boundary with no obvious impact. Applications running across this duplex mismatch will experience slower network behavior because of the TCP retries, but the user may not notice the difference, especially for a lightly loaded 100 Mbps faster connection.
However, when a real-time application is introduced, such as voice or video conferencing, the problem becomes apparent quickly. Voice and video conferencing use UDP because of their real-time nature. UDP packets that are lost as they cross this half duplex boundary are not recovered and so they cause audio or video degradation. Network personnel are often confused, because the data applications appear to be running correctly, whereas the new voice or video application appears to fail, and often the blame is put on the voice or video equipment.
Identifying a duplex mismatch: A duplex mismatch can often be recognized because it causes asymmetric packet loss and because the packet loss is consistent. Asymmetric packet loss means that the packet loss only occurs in one direction. Statistics in the video conferencing or voice endpoint will indicate packet loss in the transmit or receive direction only. If packet loss is consistently observed in the one direction but not the other, a duplex mismatch may be the reason. The other indicator is packet loss consistency. Congestion packet loss only occurs when the network is busy, and so can often be recognized by the time of day or by knowing what other traffic is occurring in the network. Duplex mismatch packet loss occurs independent of the traffic volume, especially for bidirectional streams like voice and video conferencing.
Isolation: If you think you have a duplex mismatch, review the layer 2 statistics along the network path that is failing. High counts in the layer 2 statistics (CRC, collisions, late collisions, runts and large packets) will indicate the presence of a duplex mismatch. If these statistics show high counts, look at the two devices on either side of the connection to determine the actual Ethernet connection status. Is one end operating full duplex and the other end half duplex? Many devices will show you how the devices configured, which may be in auto mode, meaning that the device will automatically negotiate the answer, but if you look deeper it will also tell you how the link is actually connected (half or full duplex). Some devices may be hardwired so that they are forced to a specific configuration, for example 100 Mbps duplex.
Resolution: My usual recommendation is to configure both sides of the connection to use auto-negotiation. In most cases, auto-negotiation will result in a common understanding of the speed and duplex for that link. If one end is forcing half or full duplex mode and the other end is using auto-negotiation, the link may fail to properly set up the duplex. The first step in resolution is to insure that both ends are configured to auto-negotiate.
If auto-negotiation on both sides is failing, then the second step is to force both sides to use the appropriate connection parameters such as 100 Mbps full-duplex. Certain combinations of vendors' equipment are subject to failing depending on how they designed the Ethernet negotiation protocol. If you find you have consistent failures in your network, then create a standard procedure within your organization to force these devices to use the right duplex choice.
The duplex mismatch problem is getting better over time as new equipment gets better negotiation testing. New equipment has had better chance of negotiating correctly. However, there is still a lot of equipment out there that will occasionally fail in the duplex negotiation and cause a packet loss headache within your network.