Ivelin Radkov - Fotolia
How to detect preinstalled malware in custom servers
Preinstalled malware was reportedly found by Apple in its custom servers. Expert Nick Lewis explains how enterprises can protect themselves from encountering similar issues.
Enterprises are under considerable cost pressure to think differently and find ways to optimize business processes at scale.
In the data center space, this has resulted in intense scrutiny regarding hardware costs, which has driven many web-scale companies to take a less traditional approach when sourcing server hardware; one in which they design most of the hardware and a custom manufacturer produces the servers.
This means that enterprises are responsible for more aspects of hardware security and rely on the custom manufacturer more than they would with a traditional server vendor. Apple appears to have ended its relationship with Super Micro Computer Inc. because of problems arising from this type of arrangement.
In this tip, we'll look at Apple's reported problems with custom servers, and what enterprises can do to protect themselves from a similar problem.
Apple's problems with custom servers
A recent report by The Information claimed that Apple removed Super Micro servers from its data centers and returned them because Apple discovered preinstalled malware in their firmware. The details on the specific malware were not reported, but it could have been a targeted attack against Apple. The malware could have been implanted in the firmware during the design or assembly stage by Super Micro or by one of Super Micro's suppliers.
Super Micro designs and assembles server components, such as network cards, storage interfaces and CPUs. For the Super Micro operating system to run on modern systems, it interfaces with the BIOS and firmware and, many times, the BIOS and firmware include significant functionality. These tools could be from an additional third-party contracted by the manufacturer. The BIOS and firmware may need to be updated, and can still be compromised.
Despite being custom servers, many of the server's components are similar to that of mass market servers, and Super Micro uses similar firmware and drivers to keep costs low. As more third parties get involved, the server attack surface grows larger, and responsibility for hardware security of the finished product gets less clear. Firmware or a BIOS can be compromised at any point, and if they are not checked by the next step in the assembly, the compromised system could make it to an enterprise's network.
While the details may be murky, the general problem of supply chain security for custom hardware is a concern for enterprises. This is similar to preinstalled malware on Android phones, preinstalled bloatware or backdoors introduced through a hardware debug environment added by the manufacturer. The National Security Agency is reported to have used similar steps.
More attention has been paid to supply chain security as a server, or any technology product could be compromised at many different points in that chain, and could arrive at the end user already compromised.
Enterprise protections for custom hardware
There are many benefits to using custom hardware, like having the option for more secure systems without unnecessary functionality and having components like trusted platform modules or cryptographic coprocessors included in the hardware. With these benefits comes the additional responsibility of checking the security of the custom servers.
Some signs that your firmware may be compromised out of the box are unexpected network connections, higher than expected CPU temperature or unexpected power usage from the malware running on the system.
The network or power can be monitored from a separate system, and unexpected activity can be investigated. If something is found, it may be possible to swap out different hardware components to identify the malicious component, but the effort to identify all the potentially compromised components can be significant. It may not be possible to identify the specific malicious component, and the only way to respond may be to completely remove the hardware, since it would otherwise be impossible to secure the system.
When bringing new servers (or anything connected to a network) into an enterprise, a standardized procedure based on the enterprise's risk tolerance should be followed. If the enterprise has a high risk tolerance, this procedure could be fairly straightforward, just involving racking and physically connecting the cabling and installing a known clean version of the desired operating system.
If an enterprise has a low risk tolerance, this procedure could be fairly detailed, involving the inspection of tamper-evident packaging to see if the packaging was modified in transit from the manufacturer, updating all of the firmware and BIOS, connecting the hardware to an unsecured network and monitoring it for any network activity, and then setting it up to see if preinstalled malware is present on the hardware.
The National Institute of Standards and Technology (NIST) has specific documentation for such scenarios, such as "BIOS Protection Guidelines for Servers." Other external standards, like NIST 800-53v4, Center for Internet Security benchmarks and Microsoft's baseline for server hardening can provide a comprehensive list of options. These standards can be adapted to meet the requirements of the enterprise to ensure that a server is deployed securely.
Conclusion
Cost and performance pressures often bring new security challenges that enterprises need to work through to ensure the appropriate protections are in place for custom servers and their supply chains. These custom servers have the potential to be more secure and less expensive than mass market servers, but require additional steps to ensure they are secure when deployed.
