Here’s a warning if you do bare metal: Some cloud providers are a tiny bit lax about cleaning servers before reassigning them to new customers.
A bare-metal server doesn’t give you the virtualization protection of a conventional public-cloud tenancy. So it’s important that any changes made to the server hardware by previous customers get fully reverted before you start using it.
Unfortunately, that’s not always been happening, leading to “critical” security issues—potentially. In this week’s Security Blogwatch, we bare all.
Your humble blogwatcher curated these bloggy bits for your entertainment. Not to mention: a brief story of hope.
BMCs laid bare
What’s the craic? Dan Goodin has Supermicro hardware weaknesses let researchers backdoor an IBM cloud server:
New research shows how baseboard management controllers … threaten premium cloud services. … BMCs are motherboard-attached microcontrollers that give extraordinary control over servers.
…
Researchers at security firm Eclypsium [published] a paper about how BMC vulnerabilities threaten … bare-metal cloud computing [which] lets customers buy exclusive access … and, when the servers are no longer needed, return them to the cloud provider. The provider, in theory, wipes the servers clean.
…
BMC vulnerabilities can undermine this model by allowing a customer to leave a backdoor that will remain active once the server is reassigned. … To prove their point, the researchers commissioned a bare-metal server from IBM's SoftLayer … made a slight modification to the BMC firmware. … The researchers then returned the server to IBM and requested new ones.
Eventually, the researchers were assigned [the same] server they had previously … modified. [The] cloud provider failed to detect and re-flash modified firmware.
Ouch. Shaun Nichols quips Don't just grin and bare it:
Cloud providers renting out bare-metal servers must make sure they scrub every last byte of writable storage on their boxes between deployments. … Otherwise, malicious customers could stash spyware and other malware in motherboard flash memory.
…
This would potentially allow an attacker … to stake out and observe specific cloud compute services to steal corporate data. In the shorter term, a scumbag could simply sabotage the firmware to brick servers en masse.
…
Eclypsium, we should point out, just so happens to sell firmware security solutions.
Oh it does, does it? Here’s the company’s blog post—The Missing Security Primer for Bare Metal Cloud Services:
BMCs have become standard components for most servers and provide management capabilities via the Intelligent Platform Management Interface (IPMI). The BMC is a highly privileged component designed to enable out-of-band management of the server.
…
IPMB channels open the door for threats to move from Internet-facing services to the underlying firmware of the host device. … Malware can potentially send malicious IPMI commands over system interfaces from the host without the commands being authenticated.
…
Since firmware underlies even the host operating system and the virtualization layers of a server, any implants would naturally be able to subvert the controls and security measures running at these higher layers. [So] any compromises to the BMC firmware can be particularly powerful for an attacker. [It] opens up the possibility for … “bricking” the server … exfiltrating data [or] ransomware attacks.
…
The capability for security-critical impact is high. … Using CVSS 3.0, we would classify it as 9.3 (Critical) Severity with the following details: CVSS:3.0/AV:L/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H.
But IBM calls this issue Low Severity:
IBM has found no indication that this vulnerability has been exploited for malicious purposes.
…
IBM has responded to this vulnerability by forcing all BMCs, including those that are already reporting up-to-date firmware, to be reflashed with factory firmware before they are re-provisioned to other customers. All logs in the BMC firmware are erased and all passwords to the BMC firmware are regenerated.
So a super-cynical Soroc tries their hand at translating IBM’s PR-speke:
We haven't really had time to look at this problem, but we have made some trivial changes so we could say we did something.
…
This stuff is hard, so we think we've done enough.
…
Besides, this is everyone's problem, and we think Eclypsium are jerks for bringing this up and attaching our name to it.
But seriously, though? Steve Ayre reckons it’s a question of trust:
You might want to read Reflections on Trusting Trust by Ken Thompson. Even if your hardware, firmware and software is open you do not know what the hardware is actually doing.
As soon as you introduce a third party you cannot trust the compiler or the manufacturer to do precisely what you asked without adding something else. And even if you bootstrap the entire system yourself from logic gates up there is still a chance that malware could infect your compiler and introduce an extra payload (as has happened in real cases).
And muhfugen takes that argument to its logical conclusion:
What good does reflashing the BMC's firmware do? The only (supported) way of doing that is from the BMC itself, and if it had been compromised, then it could just ignore the new firmware and report a successful reflash.
I really doubt they're removing the EEPROM/flash from the motherboard and flashing it and resoldering it.
Wait. Pause. Did someone say “Supermicro”? Wasn’t that the vendor implicated in the Bloomberg Businessweek spy story? Raineer rains on your parade: [You’re fired—Ed.]
This is sufficiently different from the Bloomberg claims.
A BMC is a known object, clearly listed on the specifications of the motherboard. … Bloomberg's story was all about a secret, tiny chip [that] wasn't listed.
However, earlyberd quickly flies off the handle:
Personal experience with Supermicro says … whoever is buying their hardware should be fairly knowledgeable. … It behooves the user to understand the hardware at least at a semi-professional level.
…
The point of the BMC is to replace hardware KVMs in favor of integrating into existing switch architecture, not to replace VNC/RDP protocols so you can remote in from anywhere. If people are misusing their BMCs you can't hold Supermicro, HP, or Dell responsible.
But why the heck do they let you reflash unsigned firmware? Ask Splynn:
When I worked for a company that created data center equipment, the BMC firmware was signed and the signature checked to boot. [But] I can also understand providers being reluctant to subject a system to multiple re-flashes because that flash lifetime affects the lifetime of the entire server.
…
We are seeing weaknesses in opsec. … The next step will be getting the BMC to do something and that a reflash doesn't change that. This is when it starts to get worrying.
And Rick Altherr—@kc8apf—told ya so:
I've ranted many times about servers being poorly architected for bare metal cloud offerings. Eclypsium has done the legwork to demonstrate one way that BMCs are a threat.
…
Google's Titan, Microsoft's Project Cerberus, and AWS's Nitro are all significant changes to commodity designs that provide the base for doing bare metal instances right. … Commodity server architecture is completely broken for this use case.
Meanwhile, what about that severity rating? Kenn White is just spitballing:
Will be interesting to watch — IBM rated the severity "low", while Eclypsium assigned a 9.3 "critical" CVE 3.
…
"Undetectable implant from one customer that persists to the next customer provisioning a cloud server" seems like a high severity issue. Particularly given a default hardware password.
…
Some providers (and CIOs) have long claimed that dedicated single tenant … infrastructure, be it bare metal or dedicated VM, was inherently more secure. Customer-accessible BMC was probably not considered in that risk.
The moral of the story?
If you use bare-metal servers, prove your provider’s proper procedures, previous to provisioning, please.
And finally
Awesome: All ten Star Wars movies in just five minutes
You have been reading Security Blogwatch by Richi Jennings. Richi curates the best bloggy bits, finest forums, and weirdest websites … so you don’t have to. Hate mail may be directed to @RiCHi or sbw@richi.uk. Ask your doctor before reading. Your mileage may vary. E&OE.
Image source: Nick Passero (cc:by)
Keep learning
Learn from your SecOps peers with TechBeacon's State of SecOps 2021 Guide. Plus: Download the CyberRes 2021 State of Security Operations.
Get a handle on SecOps tooling with TechBeacon's Guide, which includes the GigaOm Radar for SIEM.
The future is security as code. Find out how DevSecOps gets you there with TechBeacon's Guide. Plus: See the SANS DevSecOps survey report for key insights for practitioners.
Get up to speed on cyber resilience with TechBeacon's Guide. Plus: Take the Cyber Resilience Assessment.
Put it all into action with TechBeacon's Guide to a Modern Security Operations Center.
