A recent move, even though just a short distance, meant I had to disconnect my AT&T U-verse modem and then reconnect it. Wouldn't you know, a few days later the Internet connection started failing, with three- to four-minute spans of complete service loss. Figures, right?
One important thing to know about AT&T U-verse service is that it is a fiber-to-the-home (FTTH) offering based on a Gigabit Passive Optical Network, or GPON, architecture. OK, not that I have that out of the way, let me describe the troubleshooting process.
Troubleshooting began when I called the AT&T support center and talked to an agent, who eventually decided to dispatch an inside technician to my house. The technician's first move was to replace the Cisco modem I was using. When I told him about the recent blackouts, he then added an external Belkin uninterruptible power supply (UPS), designed specifically for the new modem. All was well... for a short while.
Within a few hours of the tech's departure, I again lost my Internet connection. This happened as the modem would either reboot or simply reconnect and then sync back up to the carrier.
AT&T like Verizon, uses a UPS with each optical network terminal (ONT) it deploys to a residence in order to "keep the light on" and, of course, the span up. In my situation, the ONT was new, just like the fiber, and a second modem still didn't garner success. What was causing the fiber to drop?
The logs simply showed the modem timing out and then reinitializing. The continuous ping from my MacBook Air revealed packets dropping intermittently:
64 bytes from 4.2.2.2: icmp_seq=6846 ttl=50 time=50.027 ms
64 bytes from 4.2.2.2: icmp_seq=6847 ttl=50 time=58.469 ms
64 bytes from 4.2.2.2: icmp_seq=6848 ttl=50 time=626.177 ms
64 bytes from 4.2.2.2: icmp_seq=6849 ttl=50 time=46.380 ms
64 bytes from 4.2.2.2: icmp_seq=6850 ttl=50 time=39.296 ms
Request timeout for icmp_seq 6851
Request timeout for icmp_seq 6852
Request timeout for icmp_seq 6853
Request timeout for icmp_seq 6854
Request timeout for icmp_seq 6855
Request timeout for icmp_seq 6856
Request timeout for icmp_seq 6857
Eventually the pings would resume, and the erratic latency would continue:
64 bytes from 4.2.2.2: icmp_seq=7038 ttl=50 time=35.405 ms
64 bytes from 4.2.2.2: icmp_seq=7039 ttl=50 time=35.624 ms
64 bytes from 4.2.2.2: icmp_seq=7040 ttl=50 time=35.605 ms
64 bytes from 4.2.2.2: icmp_seq=7041 ttl=50 time=36.135 ms
64 bytes from 4.2.2.2: icmp_seq=7042 ttl=50 time=35.749 ms
64 bytes from 4.2.2.2: icmp_seq=7043 ttl=50 time=423.246 ms
64 bytes from 4.2.2.2: icmp_seq=7044 ttl=50 time=36.107 ms
64 bytes from 4.2.2.2: icmp_seq=7045 ttl=50 time=36.318 ms
64 bytes from 4.2.2.2: icmp_seq=7046 ttl=50 time=115.176 ms
64 bytes from 4.2.2.2: icmp_seq=7047 ttl=50 time=36.825 ms
64 bytes from 4.2.2.2: icmp_seq=7048 ttl=50 time=36.770 ms
64 bytes from 4.2.2.2: icmp_seq=7049 ttl=50 time=36.395 ms
64 bytes from 4.2.2.2: icmp_seq=7050 ttl=50 time=36.440 ms
64 bytes from 4.2.2.2: icmp_seq=7051 ttl=50 time=36.226 ms
64 bytes from 4.2.2.2: icmp_seq=7052 ttl=50 time=35.571 ms
64 bytes from 4.2.2.2: icmp_seq=7053 ttl=50 time=39.528 ms
64 bytes from 4.2.2.2: icmp_seq=7054 ttl=50 time=36.201 ms
64 bytes from 4.2.2.2: icmp_seq=7055 ttl=50 time=42.419 ms
64 bytes from 4.2.2.2: icmp_seq=7056 ttl=50 time=36.756 ms
64 bytes from 4.2.2.2: icmp_seq=7057 ttl=50 time=568.390 ms
64 bytes from 4.2.2.2: icmp_seq=7058 ttl=50 time=36.421 ms
64 bytes from 4.2.2.2: icmp_seq=7059 ttl=50 time=36.365 ms
64 bytes from 4.2.2.2: icmp_seq=7060 ttl=50 time=86.979 ms
64 bytes from 4.2.2.2: icmp_seq=7061 ttl=50 time=35.505 ms
64 bytes from 4.2.2.2: icmp_seq=7062 ttl=50 time=599.214 ms
64 bytes from 4.2.2.2: icmp_seq=7063 ttl=50 time=35.592 ms
64 bytes from 4.2.2.2: icmp_seq=7064 ttl=50 time=36.211 ms
64 bytes from 4.2.2.2: icmp_seq=7065 ttl=50 time=35.823 ms
64 bytes from 4.2.2.2: icmp_seq=7066 ttl=50 time=436.547 ms
64 bytes from 4.2.2.2: icmp_seq=7067 ttl=50 time=36.287 ms
64 bytes from 4.2.2.2: icmp_seq=7068 ttl=50 time=36.313 ms
Walking through the neighborhood, I spotted an AT&T truck and struck up a conversation with the technician. He happened to be an outside fiber tech, whereas the guy who did my service call was an indoor tech. After explaining my problem, he said, "Let's go look at your ONT and test the fiber."
Using an optical time-domain reflectometer, he tested the optical fiber and didn't find anything wrong with it. He then called in, and spoke with his boss. They asked if I could help isolate the issue by disconnecting the Ethernet connections, which I did. Everything else was served by the onboard Wi-Fi. After the Internet continued to fail, I noted the modem state and saw that only the power light stayed on.
The AT&T outside tech called to inquire about the status. Hearing I was still having problems, he came back to my residence -- and brought his boss with him. We talked about the possibility of a defective ONT, but because the retail appliance doesn't have the same statistics as a managed switch or appliance, we had no way of knowing for sure if that was the case. Even so, AT&T replaced the ONT and reworked and retested the fiber.
Within a few hours the Internet connection once again failed. When I called the AT&T fiber tech to inform him, he decided to have a central office tech check the other end. But, again, the Internet connection continued to fail after a short period of time, with the modem rebooting or just re-synching up in a few minutes.
Next the fiber tech dropped off a surge protector, which I installed in front of the UPS. Again, the Internet failed several times and never with any pattern.
This time, the AT&T fiber tech unlatched the ONT so I could open the cover and view the status lights whenever the Internet failed. In doing so, I discovered that the "DATA" light was off -- meaning, the link failure wasn't the fiber but the Ethernet connection to the modem (as shown at right).
The next AT&T inside technician dispatched was assigned the task of replacing the modem again, this time with a new Motorola model. This unit contains a small lithium battery that snaps into the bottom of the unit and eliminates the need for the UPS brick I previously needed.
The Internet connection has been working since then, with no erratic ping times, either.
The AT&T techs commented that they really appreciated me working with them on this, and that they really wanted to resolve the issue. Not once did any of these techs tell me that if he found an inside customer cause to the problem that AT&T would bill me. And, in fact, each tech went out of his way to help me, even spending time to examine the logs and ping results I gathered.
I believe the original modem was disrupted by the blackout since the power went down a couple of times in succession before being restored. The techs told me the second modem was inferior. The third modem is the replacement AT&T is using for new installations going forward.
The service and level of customer service that all the AT&T members exhibited were exemplary. The fact that they brought out a new surge protector and a new and improved modem with an onboard lithium battery reveals that they understand potential issues caused by bad power and, more importantly, that they value customers. FTTH or fiber to the curb is not a cure-all solution, but if AT&T advances its footprint and provides the level of service I experienced, I feel confident that the new era of telecommunications will be better, faster and, with improvements and patience, one day maybe even easier to troubleshoot.
Follow Matt Brunk on Twitter and Google+!
@telecomworx
Matt Brunk on Google+
