Matt Brunk
Matt Brunk has worked in past roles as director of IT for a multisite health care firm; president of Telecomworx,...
Read Full Bio >>

Matt Brunk | July 17, 2015 |


Hat Off to AT&T, Unexpected Residential Partner

Hat Off to AT&T, Unexpected Residential Partner Here's a case of support techs going above and beyond in getting to the root of and fixing a connectivity problem, and making sure the customer feels valued.

Here's a case of support techs going above and beyond in getting to the root of and fixing a connectivity problem, and making sure the customer feels valued.

A recent move, even though just a short distance, meant I had to disconnect my AT&T U-verse modem and then reconnect it. Wouldn't you know, a few days later the Internet connection started failing, with three- to four-minute spans of complete service loss. Figures, right?

One important thing to know about AT&T U-verse service is that it is a fiber-to-the-home (FTTH) offering based on a Gigabit Passive Optical Network, or GPON, architecture. OK, not that I have that out of the way, let me describe the troubleshooting process.

Troubleshooting began when I called the AT&T support center and talked to an agent, who eventually decided to dispatch an inside technician to my house. The technician's first move was to replace the Cisco modem I was using. When I told him about the recent blackouts, he then added an external Belkin uninterruptible power supply (UPS), designed specifically for the new modem. All was well... for a short while.

Within a few hours of the tech's departure, I again lost my Internet connection. This happened as the modem would either reboot or simply reconnect and then sync back up to the carrier.

AT&T like Verizon, uses a UPS with each optical network terminal (ONT) it deploys to a residence in order to "keep the light on" and, of course, the span up. In my situation, the ONT was new, just like the fiber, and a second modem still didn't garner success. What was causing the fiber to drop?

The logs simply showed the modem timing out and then reinitializing. The continuous ping from my MacBook Air revealed packets dropping intermittently:

64 bytes from icmp_seq=6846 ttl=50 time=50.027 ms
64 bytes from icmp_seq=6847 ttl=50 time=58.469 ms
64 bytes from icmp_seq=6848 ttl=50 time=626.177 ms
64 bytes from icmp_seq=6849 ttl=50 time=46.380 ms
64 bytes from icmp_seq=6850 ttl=50 time=39.296 ms
Request timeout for icmp_seq 6851
Request timeout for icmp_seq 6852
Request timeout for icmp_seq 6853
Request timeout for icmp_seq 6854
Request timeout for icmp_seq 6855
Request timeout for icmp_seq 6856
Request timeout for icmp_seq 6857

Eventually the pings would resume, and the erratic latency would continue:

64 bytes from icmp_seq=7038 ttl=50 time=35.405 ms
64 bytes from icmp_seq=7039 ttl=50 time=35.624 ms
64 bytes from icmp_seq=7040 ttl=50 time=35.605 ms
64 bytes from icmp_seq=7041 ttl=50 time=36.135 ms
64 bytes from icmp_seq=7042 ttl=50 time=35.749 ms
64 bytes from icmp_seq=7043 ttl=50 time=423.246 ms
64 bytes from icmp_seq=7044 ttl=50 time=36.107 ms
64 bytes from icmp_seq=7045 ttl=50 time=36.318 ms
64 bytes from icmp_seq=7046 ttl=50 time=115.176 ms
64 bytes from icmp_seq=7047 ttl=50 time=36.825 ms
64 bytes from icmp_seq=7048 ttl=50 time=36.770 ms
64 bytes from icmp_seq=7049 ttl=50 time=36.395 ms
64 bytes from icmp_seq=7050 ttl=50 time=36.440 ms
64 bytes from icmp_seq=7051 ttl=50 time=36.226 ms
64 bytes from icmp_seq=7052 ttl=50 time=35.571 ms
64 bytes from icmp_seq=7053 ttl=50 time=39.528 ms
64 bytes from icmp_seq=7054 ttl=50 time=36.201 ms
64 bytes from icmp_seq=7055 ttl=50 time=42.419 ms
64 bytes from icmp_seq=7056 ttl=50 time=36.756 ms
64 bytes from icmp_seq=7057 ttl=50 time=568.390 ms
64 bytes from icmp_seq=7058 ttl=50 time=36.421 ms
64 bytes from icmp_seq=7059 ttl=50 time=36.365 ms
64 bytes from icmp_seq=7060 ttl=50 time=86.979 ms
64 bytes from icmp_seq=7061 ttl=50 time=35.505 ms
64 bytes from icmp_seq=7062 ttl=50 time=599.214 ms
64 bytes from icmp_seq=7063 ttl=50 time=35.592 ms
64 bytes from icmp_seq=7064 ttl=50 time=36.211 ms
64 bytes from icmp_seq=7065 ttl=50 time=35.823 ms
64 bytes from icmp_seq=7066 ttl=50 time=436.547 ms
64 bytes from icmp_seq=7067 ttl=50 time=36.287 ms
64 bytes from icmp_seq=7068 ttl=50 time=36.313 ms

Walking through the neighborhood, I spotted an AT&T truck and struck up a conversation with the technician. He happened to be an outside fiber tech, whereas the guy who did my service call was an indoor tech. After explaining my problem, he said, "Let's go look at your ONT and test the fiber."

Using an optical time-domain reflectometer, he tested the optical fiber and didn't find anything wrong with it. He then called in, and spoke with his boss. They asked if I could help isolate the issue by disconnecting the Ethernet connections, which I did. Everything else was served by the onboard Wi-Fi. After the Internet continued to fail, I noted the modem state and saw that only the power light stayed on.

The AT&T outside tech called to inquire about the status. Hearing I was still having problems, he came back to my residence -- and brought his boss with him. We talked about the possibility of a defective ONT, but because the retail appliance doesn't have the same statistics as a managed switch or appliance, we had no way of knowing for sure if that was the case. Even so, AT&T replaced the ONT and reworked and retested the fiber.

Within a few hours the Internet connection once again failed. When I called the AT&T fiber tech to inform him, he decided to have a central office tech check the other end. But, again, the Internet connection continued to fail after a short period of time, with the modem rebooting or just re-synching up in a few minutes.

Next the fiber tech dropped off a surge protector, which I installed in front of the UPS. Again, the Internet failed several times and never with any pattern.


This time, the AT&T fiber tech unlatched the ONT so I could open the cover and view the status lights whenever the Internet failed. In doing so, I discovered that the "DATA" light was off -- meaning, the link failure wasn't the fiber but the Ethernet connection to the modem (as shown at right).

The next AT&T inside technician dispatched was assigned the task of replacing the modem again, this time with a new Motorola model. This unit contains a small lithium battery that snaps into the bottom of the unit and eliminates the need for the UPS brick I previously needed.

The Internet connection has been working since then, with no erratic ping times, either.

The AT&T techs commented that they really appreciated me working with them on this, and that they really wanted to resolve the issue. Not once did any of these techs tell me that if he found an inside customer cause to the problem that AT&T would bill me. And, in fact, each tech went out of his way to help me, even spending time to examine the logs and ping results I gathered.

I believe the original modem was disrupted by the blackout since the power went down a couple of times in succession before being restored. The techs told me the second modem was inferior. The third modem is the replacement AT&T is using for new installations going forward.

The service and level of customer service that all the AT&T members exhibited were exemplary. The fact that they brought out a new surge protector and a new and improved modem with an onboard lithium battery reveals that they understand potential issues caused by bad power and, more importantly, that they value customers. FTTH or fiber to the curb is not a cure-all solution, but if AT&T advances its footprint and provides the level of service I experienced, I feel confident that the new era of telecommunications will be better, faster and, with improvements and patience, one day maybe even easier to troubleshoot.

Follow Matt Brunk on Twitter and Google+!
Matt Brunk on Google+


May 23, 2018

If Cisco is a critical strategic supplier for you, then its vital for you to understand the most recent announcements around Webex Teams, and how to build your migration path for the next generatio

April 25, 2018

Yesterdays simple phone call to a contact center or business location is rapidly becoming todays interactive session, spanning multiple channels and devices. If you need to know how Omnichannel can

March 7, 2018

Video collaboration is experiencing significant change and innovation-how can your enterprise take advantage? In this webinar, leading industry analyst Ira Weinstein will present detailed analysis

March 12, 2018
An effective E-911 implementation doesn't just happen; it takes a solid strategy. Tune in for tips from IT expert Irwin Lazar, of Nemertes Research.
March 9, 2018
IT consultant Steve Leaden lays out the whys and how-tos of getting the green light for your convergence strategy.
March 7, 2018
In advance of his speech tech tutorial at EC18, communications analyst Jon Arnold explores what voice means in a post-PBX world.
February 28, 2018
Voice engagement isn't about a simple phone call any longer, but rather a conversational experience that crosses from one channel to the next, as Daniel Hong, a VP and research director with Forrester....
February 16, 2018
What trends and technologies should you be up on for your contact center? Sheila McGee-Smith, Contact Center & Customer Experience track chair for Enterprise Connect 2018, gives us the lowdown.
February 9, 2018
Melanie Turek, VP of connected work research at Frost & Sullivan, walks us through key components -- and sticking points -- of customer-oriented digital transformation projects.
February 2, 2018
UC consultant Marty Parker has crunched lots of numbers evaluating UC options; tune in for what he's learned and tips for your own analysis.
January 26, 2018
Don't miss out on the fun! Organizer Alan Quayle shares details of his pre-Enterprise Connect hackathon, TADHack-mini '18, showcasing programmable communications.
December 20, 2017
Kevin Kieller, partner with enableUC, provides advice on how to move forward with your Skype for Business and Teams deployments.
December 20, 2017
Zeus Kerravala, principal analyst with ZK Research, shares his perspective on artificial intelligence and the future of team collaboration.
December 20, 2017
Delanda Coleman, Microsoft senior marketing manager, explains the Teams vision and shares use case examples.
November 30, 2017
With a ruling on the FCC's proposed order to dismantle the Open Internet Order expected this month, communications technology attorney Martha Buyer walks us through what's at stake.
October 23, 2017
Wondering which Office 365 collaboration tool to use when? Get quick pointers from CBT Nuggets instructor Simona Millham.
September 22, 2017
In this podcast, we explore the future of work with Robert Brown, AVP of the Cognizant Center for the Future of Work, who helps us answer the question, "What do we do when machines do everything?"
September 8, 2017
Greg Collins, a technology analyst and strategist with Exact Ventures, delivers a status report on 5G implementation plans and tells enterprises why they shouldn't wait to move ahead on potential use ....
August 25, 2017
Find out what business considerations are driving the SIP trunking market today, and learn a bit about how satisfied enterprises are with their providers. We talk with John Malone, president of The Ea....
August 16, 2017
World Vision U.S. is finding lots of goodness in RingCentral's cloud communications service, but as Randy Boyd, infrastructure architect at the global humanitarian nonprofit, tells us, he and his team....
August 11, 2017
Alicia Gee, director of unified communications at Sutter Physician Services, oversees the technical team supporting a 1,000-agent contact center running on Genesys PureConnect. She catches us up on th....
August 4, 2017
Andrew Prokop, communications evangelist with Arrow Systems Integration, has lately been working on integrating enterprise communications into Internet of Things ecosystems. He shares examples and off....
July 27, 2017
Industry watcher Elka Popova, a Frost & Sullivan program director, shares her perspective on this acquisition, discussing Mitel's market positioning, why the move makes sense, and more.
July 14, 2017
Lantre Barr, founder and CEO of Blacc Spot Media, urges any enterprise that's been on the fence about integrating real-time communications into business workflows to jump off and get started. Tune and....
June 28, 2017
Communications expert Tsahi Levent-Levi, author of the popular blog, keeps a running tally and comprehensive overview of communications platform-as-a-service offerings in his "Choosing a W....
June 9, 2017
If you think telecom expense management applies to nothing more than business phone lines, think again. Hyoun Park, founder and principal investigator with technology advisory Amalgam Insights, tells ....
June 2, 2017
Enterprises strategizing on mobility today, including for internal collaboration, don't have the luxury of learning as they go. Tony Rizzo, enterprise mobility specialist with Blue Hill Research, expl....
May 24, 2017
Mark Winther, head of IDC's global telecom consulting practice, gives us his take on how CPaaS providers evolve beyond the basic building blocks and address maturing enterprise needs.
May 18, 2017
Diane Myers, senior research director at IHS Markit, walks us through her 2017 UC-as-a-service report... and shares what might be to come in 2018.
April 28, 2017
Change isn't easy, but it is necessary. Tune in for advice and perspective from Zeus Kerravala, co-author of a "Digital Transformation for Dummies" special edition.
April 20, 2017
Robin Gareiss, president of Nemertes Research, shares insight gleaned from the firm's 12th annual UCC Total Cost of Operations study.
March 23, 2017
Tim Banting, of Current Analysis, gives us a peek into what the next three years will bring in advance of his Enterprise Connect session exploring the question: Will there be a new model for enterpris....
March 15, 2017
Andrew Prokop, communications evangelist with Arrow Systems Integration, discusses the evolving role of the all-important session border controller.
March 9, 2017
Organizer Alan Quayle gives us the lowdown on programmable communications and all you need to know about participating in this pre-Enterprise Connect hackathon.
March 3, 2017
From protecting against new vulnerabilities to keeping security assessments up to date, security consultant Mark Collier shares tips on how best to protect your UC systems.
February 24, 2017
UC analyst Blair Pleasant sorts through the myriad cloud architectural models underlying UCaaS and CCaaS offerings, and explains why knowing the differences matter.
February 17, 2017
From the most basics of basics to the hidden gotchas, UC consultant Melissa Swartz helps demystify the complex world of SIP trunking.
February 7, 2017
UC&C consultant Kevin Kieller, a partner at enableUC, shares pointers for making the right architectural choices for your Skype for Business deployment.
February 1, 2017
Elka Popova, a Frost & Sullivan program director, shares a status report on the UCaaS market today and offers her perspective on what large enterprises need before committing to UC in the cloud.
January 26, 2017
Andrew Davis, co-founder of Wainhouse Research and chair of the Video track at Enterprise Connect 2017, sorts through the myriad cloud video service options and shares how to tell if your choice is en....
January 23, 2017
Sheila McGee-Smith, Contact Center/Customer Experience track chair for Enterprise Connect 2017, tells us what we need to know about the role cloud software is playing in contact centers today.