- Verizon blamed the outage on a software issue
- Roger Entner said it was down to an update on the 5G SA core
- Is this goodbye to five-nines reliability on the network?
Verizon has blamed its massive nationwide outage last week on a "software issue." Analysts and commentators are weighing in on what might have happened, but the present cloud and software code-based networks of major mobile operators make such outages likely to happen again.
Roger Entner, founder and analyst at Recon Analytics, noted that the hours long outage likely affected around 1.5 million Verizon subscribers. "It was likely a minor software update on the 5G standalone core that went wrong in the middle of the day,” he said directly on LinkedIn.
“The most likely failure domain wasn’t access (cell towers or fiber lines), but shared backend systems,” suggested Arslaan Khan, a solutions architect, on LinkedIn. The outage was a control plane problem, he said.
“When these systems fail, radios still transmit and fiber stays lit — but devices can't authenticate or establish sessions, impacting both mobile subscribers and home broadband users,” Khan said. Like other commentators, he suggested that the software-based cellular network systems enable scale, but without barriers can permit “fast, nationwide failure.”
Fierce spoke to Sanjoy Paul, PhD, an associate vice president and lecturer in the computer science department at Rice University in Houston, Texas, late last week. Paul has worked in his career on building 2G, 3G and 4G networks and helped in building a 5G-in-a-box system.
He said that modern cellular networks — like those from Verizon — are software-based and take a lot of regular code updates. “In the 5G standalone network, it’s a virtualized core, so you have to send out patches upon configuration, if there’s a problem in that, it could also lead to problems,
“When you open up your cellphone and you want to connect to the internet through the cellular network, there are about 20 signaling messages that happen even before you get to send the first bit through the wireless network,” he said. “Out of those 20 messages, if one of them has even a little problem the connection will not happen.”
Say goodbye to five-nines
One thing that Paul noted with the move to software-based cellular networks was that the concept of five-nines reliability is out the window.
Previously, he said, operators were slow to provision services on the network and it cost a lot of money, but they could offer five-nine reliability.
Paul noted that five-nines would mean that a telecom network would be down for at most 5 minutes and 15 seconds a year. This is simply not the case any more.
With operators moving to a software-based provisioning system, reliability is more like three-nines, which is 8 hours and 30 minutes down throughout the year, he said. “The outage was 10 hours for Verizon, so three-nines was not even met,” Paul stated.
As previous outages from all three major mobile operators in the U.S. have averaged approximately 10 hours, the five-nines concept may well be dead.