Forget Smurf, Fraggle, and DNS amplification attacks. This one’s HUGE.
With total traffic approaching 1Tbit/sec., misuse of badly configured memcache daemons is causing havoc all over the Internet. And it appears this is just the tip of the iceberg.
Don’t be part of the problem: Be part of the solution. In this week’s Security Blogwatch, we make like Nigel Tufnel.
Your humble blogwatcher curated these bloggy bits for your entertainment. Not to mention: Uncertainty …
Amps up to 11...
There’s a craic, and Dan Goodin cleverly cleaves it—In-the-wild DDoSes use new way to achieve unthinkable sizes:
Hackers have found a way to amplify distributed denial-of-service attacks by an unprecedented 51,000 times … making it by far the biggest amplification method ever used in the wild. [The] development … could lead to new record-setting assaults [on] websites and Internet infrastructure.
…
DDoS vandals have long [amplified] their attacks by sending a small number of specially designed data packets to publicly available services. The services then unwittingly respond by sending a much larger [reply]. The vector this time is memcached, a database caching system for speeding up websites and networks. Over the past week, attackers have started abusing it to deliver DDoSes with volumes of 500 gigabits per second and bigger.
…
The attacks work because … networks [are] exposing memcached servers to the Internet in their default unsecured configuration. Generally speaking, memcached systems should be reachable only on local networks and should be kept securely behind a firewall.
So far, attacks have come from slightly more than 5,700 unique IP addresses, mostly in North America and Europe. [But there are] more than 88,000 poorly secured memcached servers, as measured by the Shodan search engine.
What is this “amplification” of which you speak? Richard Chirgwin explains—Get it behind the firewall and turn off UDP if you want to live:
The mechanism [is] to send memcached instances a request for statistics over UDP, apparently coming from the spoofed victim's IP address. The stats request is 15 bytes long, but the reply is … up to hundreds of kilobytes. … If you're under attack, there are two things to do: block all traffic from port 11211, and if you can, get help from your ISP to block the traffic.
…
And if you're a sysadmin whose memcached server is outside the firewall, get it inside, configure it so it doesn't listen on UDP, and strap yourself to the butt-kicking machine, because [its] config file told you not to put it on the Internet.
Yeah, but nobody can be expected to read the instructions. Kevin Liston lists a fix: [You’re fired—Ed.]
But who has time for instructions, right? What's Google tell me to do? The top answer gives us:
…
Change the memcached configuration setting for CACHESIZE and -l :
1. Open /etc/memcached.conf in a text editor
2. Locate the -m parameter
3. Change its value to at least 1GB
4. Locate the -l parameter
5.Change its value to 127.0.0.1 …
6. Save your changes to memcached.conf …
7. Restart memcached
Digital Ocean's instructions … looks fine too. So it's not a rash of bad instructions on the Internet.
Wait, surely a new vuln should have a sexy name? Marek Majkowski obliges—Memcrashed:
Over last couple of days we've seen a big increase in an obscure amplification attack vector [on] UDP port 11211. … The general idea behind all amplification attacks is the same. An IP-spoofing … attacker sends forged requests to a vulnerable UDP server. The UDP server, not knowing the request is forged, politely prepares the response [but] the response packets are much larger than the request.
…
This is massive for a new amplification vector. But the numbers don't lie. It's possible because all the reflected packets are very large.
…
I was surprised to learn that memcached does UDP, but there you go! The protocol specification shows that it's one of the best protocols to use for amplification ever! There are absolutely zero checks, and the data WILL be delivered to the client, with blazing speed! Furthermore, the request can be tiny and the response huge (up to 1MB).
…
If you are using memcached, please disable UDP support if you are not using it. [Or] ensure that your memcached servers are firewalled from the internet!
…
Developers: Please please please stop using UDP. If you must, please don't enable it by default. [And] remember that people do forget to set up a firewall. Be a nice citizen. Don't invent a UDP-based protocol that lacks authentication.
Time to panic? Barry Greene has advice for All Operators and Enterprise Networks:
Operators are asked to review their networks and consider updating their Exploitable Port Filters (Infrastructure ACLs) to track or block UDP/TCP port 11211 for all ingress and egress traffic.
…
Enterprises are asked to update their iACLs, Exploitable Port Filters, and Firewalls to track or block UDP/TCP port 11211 for all ingress and egress traffic.
Can we have more detail? Roland Dobbins and Steinthor Bjarnason assert these Mitigation Recommendations:
memcached is an in-memory database caching system which is typically deployed in [data center] … networks to improve the performance of … Internet-facing services. … Unfortunately, there are many memcached deployments worldwide which have been deployed using the default insecure configuration, and without benefit of situationally-appropriate network access policies.
…
It is critical that network operators take proactive measures to ensure they are prepared to detect, classify, traceback, and mitigate these attacks. … Attacks can be successfully mitigated by implementing industry-standard Best Current Practices (BCPs) such as source-address validation/BCP38/BCP84; by leveraging network infrastructure functionality such as flowspec, transit ACLs (tACLs), and selective quality-of-service (QoS) policies; and by utilizing intelligent DDoS mitigation systems.
…
In particular, state minimization is highly encouraged as a general operational principle to increase resilience in the face of attack. Situationally-appropriate network access policies should be implemented via … tACLs on … upstream transit links to block unauthorized network traffic destined for UDP/11211. … Given that intentional production use of memcached across the public Internet is vanishingly rare, traffic sourced from UDP/11211 may be safely rate-limited at peering/transit/customer aggregation edges.
What about some firsthand accounts? April Linden pops up from her eponymous Labs:
As I’m sure most of y’all have noticed, Second Life has had a rough 24 hours. We’re experiencing outages unlike any in recent history.
…
Second Life being hit with a DDoS attack is pretty routine. … However, the current DDoS attacks are at a level that we rarely see.
…
Second Life is not the only Internet service that’s been targeted today. … It’s been a rough few days on much of the Internet.
…
See you inworld!
And from the amplifying end, here’s Matthew “muglug” Brown:
I was caught up in this, but luckily my hosting provider (MediaTemple) disabled my server due to the large bandwidth overruns.
…
It was fun to spend Sunday night hurriedly reading up on iptables. Still getting a lot of inbound traffic (everything is getting dropped), but thankfully no outbound now.
Let those who are without sin chuck the first rocks. Respect Deputy Cartman’s authority:
This just reeks of companies that simply won't hire enough people to provide the "brain cycles" to setup proper infrastructure, so people are overworked and make mistakes, or corporate culture allows for who-you-know mattering more than what-you-know.
But hey, DDOS attacks on others don't show up on our quarterly results so who cares, right?!
And Brian Miller wonders when people will start listening to security people:
Too many numpties, too many deadlines.
When you have a combination of incompetent managers and developers, these things happen all the time. From IOT to big servers, and sometimes it happens with the security dev screaming bloody murder. "Nah, nobody is going to do that..."
But jdw casts the blame net wider:
Not only are the publicly-exposed memcached servers irresponsible, but [so is] the continued willingness of Internet backbone providers to transit forged packets.
This is hardly insurmountable. Less than ten companies working together could effectively end many types of DDOS attacks tomorrow if they would [all] simply refuse to ingress forged packets from their customers. This is already a best-practice configuration for border routers. … They just don't do it.
Meanwhile, Clint “@green_clint” Green waxes sardonic:
I like to imagine it was like "take a penny, leave a penny." They were left accessible to help performance on anyone's DB...
…
It's just like any of the 0.0.0.0 defaults you can look for in Shodan. With great power (cloud, scale) comes great responsibility (safe defaults, understanding tools that are deployed).
Risk in public becomes shared risk with no op out.
The moral of the story? Make sure all your internal-only services are only accessible internally. And if you route traffic, consider filtering packets with impossible source IPs.
And finally …
Are you uncertain about quantum physics? This video explains all (or doesn’t)
You have been reading Security Blogwatch by Richi Jennings. Richi curates the best bloggy bits, finest forums, and weirdest websites … so you don’t have to. Hate mail may be directed to @RiCHi or sbw@richi.uk. Ask your doctor before reading. Your mileage may vary. E&OE.
Image source: jboylan67 (cc:by)
Keep learning
Learn from your SecOps peers with TechBeacon's State of SecOps 2021 Guide. Plus: Download the CyberRes 2021 State of Security Operations.
Get a handle on SecOps tooling with TechBeacon's Guide, which includes the GigaOm Radar for SIEM.
The future is security as code. Find out how DevSecOps gets you there with TechBeacon's Guide. Plus: See the SANS DevSecOps survey report for key insights for practitioners.
Get up to speed on cyber resilience with TechBeacon's Guide. Plus: Take the Cyber Resilience Assessment.
Put it all into action with TechBeacon's Guide to a Modern Security Operations Center.
