Eclypsium: Bare-metal cloud servers vulnerable to firmware attacks

Threat actors can use firmware attacks on bare-metal cloud servers to easily gain persistent access to the hardware, according to new research from hardware security startup Eclypsium.

The research showed how vulnerabilities in baseboard management controllers (BMCs) and weaknesses in the reclamation process of bare-metal cloud servers can allow attackers to add other malicious implants that can persist and steal data. Eclypsium researchers tested the attack scenario on IBM's SoftLayer cloud services and found the vulnerability, which it calls Cloudborne, existed on SoftLayer's bare-metal services. But Eclypsium said such firmware attacks are almost certainly possible on other cloud services.

"This is an industrywide issue, rather than just an IBM issue," said John Loucaides, vice president engineering at Eclypsium, based in Beaverton, Ore. "IBM is a case study, but IBM is certainly big enough that they have real security teams that are doing this sort of research, and they were able to miss this. That means all the other folks are capable of missing this, as well."

Eclypsium researchers claimed the firmware attacks are easy to execute and don't require any sort of significant hacking skills. Cloudborne can affect premium cloud services and leave customers open to a variety of threats, including implanted backdoors, the research claimed.

"Bare-metal offers you a lot of interesting capabilities, and you would think, because it gives you more control, it would give you a stronger security posture. But what we found was that, in fact, it gives the adversary [or] the attacker more control, as well," Loucaides said. "This type of attack is something that could apply both to bare-metal cloud and regular cloud, but the obvious application is on bare-metal, where you have direct access to the hardware."

As bare-metal cloud offerings are used for sensitive applications, bare-metal hardware is dedicated to one customer at a time. As deployments end, hardware is reclaimed by the cloud service provider and is reprovisioned to the next customer. With hardware vulnerable to Cloudborne, threat actors can make simple changes to the firmware.

With multiple tenants using the same resources over time, Loucaides said, cloud providers need to sanitize those resources in between giving it from one user to another.

"What we found [in our research] was that they were missing the sanitization of the firmware components, and that leaves you vulnerable to attacks," he said.

When dealing with bare-metal clouds, Loucaides suggested the simple thing to do would be to just reinstall or update the firmware image of the different components, particularly the BMC.

BMCs have become standard components for most servers and provide management capabilities via the Intelligent Platform Management Interface (IPMI), Eclypsium said.

"The BMC provides the out-of-band management interface. So, it has a lot of power over the system, and it's one of the critical components that you would want to make sure that you bring back to a known state in between de-provisioning a system from one user and provisioning it to another," Loucaides added.

IBM SoftLayer case study

To test these firmware attacks, Eclypsium researchers rented a bare-metal server from IBM's SoftLayer cloud services. Researchers also noticed the server was using a BMC from Supermicro, a hardware vendor with known firmware vulnerabilities.

After confirming it had the latest BMC firmware available, researchers recorded the chassis and product serial numbers to help them identify the system later. The research team then made a "benign change" to the BMC firmware in the form of a single bit flip. An additional user account in the BMC's IPMI was also created before releasing the server back to IBM.

Researchers then reacquired the same piece of hardware and found that while the additional IPMI user was removed, the BMC firmware containing the flipped bit was still present.

If you look at the capabilities that this offers you, it offers you a way to persist a piece of malware from one tenant to another.

This indicated the servers' BMC firmware was not reflashed during the server reclamation process, according to the research. The combination of using vulnerable hardware and not reflashing the firmware makes it possible to implant malicious code into the server's BMC firmware, researchers concluded.

Researchers also noticed BMC logs were retained across provisioning, and the BMC root password remained the same across provisioning.

"If you look at the capabilities that this offers you, it offers you a way to persist a piece of malware from one tenant to another," Loucaides said. "The obvious things that a hacker will want to do will include stealing data and exfiltrating some secret information from the other tenant. Another interesting one is the idea of providing a substantial disruption to the infrastructure by effectively bringing down those machines. If you have access at this firmware layer, you can permanently 'brick' a machine."

Loucaides suggested customers and cloud service providers should ensure security at the firmware level. For example, just monitoring that layer is very useful, he said.

"Even if you don't take action, like deliberately installing a particular firmware version, just checking to see if something changed is a good mechanism to know whether or not you might have a problem," he said.

Loucaides emphasized that Cloudborne can affect many cloud providers and should not be considered limited to IBM SoftLayer. He said there are a lot of smaller players that are going to have a much harder time dealing with this and understanding this.

IBM has responded to this vulnerability by forcing all BMCs, including those that are already reporting up-to-date firmware, to be reflashed with factory firmware before they are reprovisioned to other customers, according to a company blog post. All logs in the BMC firmware were erased, and all passwords to the BMC firmware were regenerated, the post added.

"We are not aware of any client or IBM data being put at risk because of this reported potential vulnerability, and we have taken actions to eliminate the vulnerability," an IBM spokesperson said. "Given the remediation steps we have taken and the level of difficulty required to exploit this vulnerability, we believe the potential impact to clients is low. While the report focuses on IBM, this was actually a potential industrywide vulnerability for all cloud service providers, and we thank Eclypsium for bringing it to the attention of the industry."

While IBM categorized this as a "low-severity" issue, Eclypsium said it does not agree with the characterization. "Using CVSS 3.0," the vendor wrote in its research paper, "we would classify it as 9.3 (critical) Severity."