Monday’s scare of a “major” denial-of-service attack turns out to have been just a BGP misconfiguration. Or some other fat-fingered change.
It only really affected T-Mobile’s core IP network. But an avalanche of misunderstanding, hype, and bad reporting turned it into “a major DDoS attack.”
And one of the biggest culprits was a huge Twitter account claiming to speak for the Anonymous collective. In this week’s Security Blogwatch, we do not forgive.
Your humble blogwatcher curated these bloggy bits for your entertainment. Not to mention: Black Box.
T reconfig FAIL
What’s the craic? Graig Graziosi reports—Problems appear to have originated on T-Mobile's network:
T-Mobile, Verizon, AT&T, Sprint all received spikes in reported outages beginning at around midday eastern time on Monday. Customers of the major carriers took to Twitter to complain that despite having full bars, they had no cell service.
…
Customers called for refunds or an application of phone credit to their accounts as compensation for the disruption to their service. One user, going by the name Middlemontana, said … "It's 2020 a major outage like this nationwide simply cannot occur. You are essentially a utility company. If you don't have redundancy you aren't prepared to be the major data provider for the US with 5G."
…
According to Down Detector, a website that monitors cell service and website outages, the carriers have collectively received more than 112,000 reports of outages today, with the majority - more than 100,000 - coming from T-Mobile customers. New York City, Texas, Florida, Georgia and California all reported significant outages.
And Anthony Garreffa adds—Hundreds of thousands of Americans hit with … outages:
If you're experiencing outages right now, you're not alone -- millions are being hit with internet, phone, and data outages across the United States. … T-Mobile was hit hard, with voice and data services affected and massive spikes in outage reports. … But it wasn't just T-Mobile: it was virtually everyone. AT&T, Verizon, Sprint — and even YouTube, Fortnite, and Twitch were majorly affected at the same time.
Oh, so not just the cell providers? Emily Bicks gets heavy, man—Anonymous Tweets U.S. Hit by Major DDoS Attack:
Following a massive cell phone service outage that affected hundreds of thousands of T-Mobile, AT&T, Verizon and Sprint customers … hacktivist group Anonymous tweeted that it was a result of a “major DDoS attack.” … Anonymous tweeted out a digital map that appeared to show the various types of attacks happening between America and the rest of the world.
…
It wasn’t just major mobile networks that reported outages. … Customers reported issues with Twitch, Comcast, Facebook and Instagram.
Waddya mean “Anonymous tweeted”? A breathless @YourAnonCentral bars no holds:
The U.S. is currently under a major DDoS attack. … All major cell phone providers across the United States are currently suffering from major outages.
…
Facebook, Instagram, and Twitch are also experiencing issues across the United States. Twitter remains reliable for now. Obviously, we need a new decentralized internet.
…
The source of the DDoS attack on the United States is currently unknown. We speculate it may be China as the situation between South and North Korea is currently deteriorating. … North Korea blew up an inter-Korean liaison office established to maintain relations between Seoul and Pyongyang on the border town of Kaesong. … The North Korean army awaiting orders to enter DMZ.
…
We are the bottom coming for the top. Global revolution now.
O RLY? Davey Winder scoffs—No, The U.S. Has Not Suffered The Biggest Cyber-Attack In History:
As hundreds of thousands of people reported mobile carriers and internet services down, and 'DDoS' started trending on Twitter, 'Anonymous' laid the blame on China. … The plot thickened as a supposed Anonymous news account with 6.5 million followers stirred the pot.
…
Pretty soon, the DDoS (Distributed Denial of Service) hashtag was trending on Twitter, and anyone experiencing any connectivity issue was blaming this [on a] major cyber-attack. [But] none of the mobile carrier networks, internet providers or online services were reporting any major downtime. Apart from one: T-Mobile.
…
People unable to connect to services, because the T-Mobile network core was disrupted, reported those services as being down. People failing to connect calls to other mobile carriers reported them as being down.
…
Retweets are not evidence, rumor is not fact. … Don't believe everything that 'Anonymous' accounts on Twitter say.
Expert analysis, plz. Matthew Prince—@eastdakota—unpicks the conundrum:
There’s a lot of buzz right now about a “massive DDoS attack” targeting the US, complete with scary-looking graphs. … While it makes for a good headline in these already dramatic times … the reality is far more boring.
…
It starts with T-Mobile. They were making some changes to their network configurations [that] went badly. The result has been for around the last 6 hours a series of cascading failures for their users, impacting both their voice and data networks.
…
That caused a lot of T-Mobile users to complain on Twitter and other forums that they weren’t able to reach popular services. Then services like Down Detector scrape Twitter and report those services as being offline.
…
So now people are looking around for an explanation and they stumble across sites like the Arbor Networks attack map. It looks terrifying today! Thing is, it always looks terrifying. It’s a marketing gimmick put up to sell DDoS mitigation services so that’s not surprising.
…
We can see a number of things that show there is no massive DDoS attack. First, traffic … to supposedly impacted services is normal. … Second, there is no spike in traffic to any of the major Internet Exchanges, which you do see during actual DDoS attacks.
But how does an IP issue affect phone calls? T-Mobile’s Mike Sievert and Neville Ray explain—Update on T-Mobile Network Issues:
This is an IP traffic related issue that has created significant capacity issues in the network core throughout. … Many of our customers experienced a voice and text issue … specifically with VoLTE (Voice over LTE) calling.
…
Data connections continued to work, as did our non-VoLTE calling for many customers. … VoLTE and text in all regions were fully recovered by 10 p.m. PDT.
…
Our engineers worked through the night to understand the root cause … address it and prevent it from happening again. The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy.
…
[But] this redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.
…
We have worked with our IMS … and IP vendors to add permanent additional safeguards to prevent this from happening again. … We’re continuing to work on determining the cause of the initial overload failure.
tl;dr? troutman’s got your back:
T-Mobile's problems are self-inflicted—from work to merge their network with Sprint.
So dump T? Maybe; maybe not. jhodge hedges: [You’re fired—Ed.]
T-Mobile customer here. … I'm not particularly worked up about this.
If it happened regularly, sure, but over the past decade my T-Mobile service has been more reliable than my electrical or water service, and far more reliable that AWS or Azure. Occasionally, people make mistakes and **** happens.
…
It's not 5 9's, but I honestly don't want to pay for that level of availability.
But what of the tweep claiming to speak for a group that isn’t a group and has no centralized spokespeople? Dan East goes west at the claims about China:
Only an imbecile would make a statement like that. Not that Anonymous had much credibility in the first place.
…
So China is going to show their hand and demonstrate their cyberwarfare capability to cause some minor inconvenience to a handful of US companies because the Koreas are having their usual spat of the month? Absurd.
Meanwhile, Nick- dashes off this lesson in learning from the mistakes of others:
Definitely some kind of BGP misconfig. Don't skip your peer reviews, guys.
The moral of the story?
Change control, peer review, and pair programming: not just good ideas, no matter how experienced you are.
And finally
You have been reading Security Blogwatch by Richi Jennings. Richi curates the best bloggy bits, finest forums, and weirdest websites … so you don’t have to. Hate mail may be directed to @RiCHi or sbw@richi.uk. Ask your doctor before reading. Your mileage may vary. E&OE. 30.
This week’s awesomesauce imagery via @YourAnonCentral.
Keep learning
Learn from your SecOps peers with TechBeacon's State of SecOps 2021 Guide. Plus: Download the CyberRes 2021 State of Security Operations.
Get a handle on SecOps tooling with TechBeacon's Guide, which includes the GigaOm Radar for SIEM.
The future is security as code. Find out how DevSecOps gets you there with TechBeacon's Guide. Plus: See the SANS DevSecOps survey report for key insights for practitioners.
Get up to speed on cyber resilience with TechBeacon's Guide. Plus: Take the Cyber Resilience Assessment.
Put it all into action with TechBeacon's Guide to a Modern Security Operations Center.
