SolarWinds struggles with response to supply chain attack

SolarWinds has faced setbacks in its response to the supply chain attack that led to the compromise of numerous enterprise and government agency customers.

The attack involved the insertion of a backdoor by nation-state threat actors into updates for SolarWinds' Orion product that was then distributed to customers worldwide. The attack was first disclosed on Sunday by FireEye, which is a SolarWinds client and was breached as a result of the implanted backdoor. The backdoor was contained in a DLL component, SolarWinds.Orion.Core.BusinessLayer.dll, that was part of SolarWinds' digitally signed updates for Orion.

SolarWinds took immediate steps Sunday to address the attack, issuing a cybersecurity advisory and urging customers to upgrade to the latest version, while releasing a new hotfix to mitigate the backdoor on Tuesday. However, issues still plagued the vendor's response.

Namely, there were multiple reports of the DLL in updates available for public download on SolarWinds' website as recent as Monday evening (the download links have since been removed). Moreover, SolarWinds has a support page that advises users to disable antivirus scans on Orion product folders.

Reports of the DLL still being available came from GreyNoise Intelligence founder Andrew Morris, as well as others like Huntress Labs CEO Kyle Hanslovan and Casey Ellis, CTO, founder and chairman of Bugcrowd. Morris said via Twitter that the backdoored DLL was still present in an installer on SolarWinds website.

In an emailed statement from Hanslovan and Huntress senior security researcher John Hammond, they explained that they found "three SolarWinds programs and 12 distinct locations on a computer's filesystem where the DLL can be present." The research also showed the DLL file was still present on SolarWinds' website.

"We discovered the backdoored DLL, SolarWinds.Orion.Core.BusinessLayer.dll, was found within several official updates from SolarWinds. For some time, there were three fully compromised packages still publicly available for download from SolarWinds' website but have since been removed after we reported the findings," the email read.

Hanslovan and Hammond noted that the presence of the DLL alone does not indicate a compromise, and there could be instances of the DLL that do not contain the backdoor code.

"The backdoored DLLs all contain one specific class of code, OrionImprovementBusinessLayer, that was added by the attackers," they said via email. "Our analysis classifies DLLs into four categories: (1) malicious -- we have verified the malicious Sunburst code is present in the OrionImprovementBusinessLayer class; (2) reputable -- we have that verified the class and malicious code is not present; (3) suspicious -- the class is present but contains no malicious functionality, and finally (4) unknown -- where we have not yet verified the status."

In an email to SearchSecurity, Ellis explained the DLL further.

"The CORE-2019.4.5220.20574-SolarWinds-Core-v2019.4.5220-Hotfix5.msp hotfix package to Orion was compromised on the SolarWinds update servers, and the SolarWinds.Orion.Core.BusinessLayer.dll DLL backdoored with the Sunburst payload. The DLL itself should be present in any SolarWinds Orion install, but if it has a hash of [b91ce2fa41029f6955bff20079468448] it is the compromised version," he said. "This DLL is highly privileged and can retrieve 'jobs' from its C2 servers. Given that Orion typically has a variety of highly privileged access credentials for the systems it monitors, this presents an opportunity for comprehensive access for an attacker."

UPDATE 12/17: Morris told SearchSecurity that while the lingering presence of the backdoored DLL was concerning, he sympathizes with SolarWind's struggles to fully contain and address a complex and sophisticated attack.

"I think it's really easy to pile on [SolarWinds] for [their response], but I know that they're doing the best that they can, and things like this are monumentally challenging to deal with. It's very easy for people to feel they're not doing enough or they're doing poorly or whatever, but until and unless you've been through a compromise of this scale and by actors of this level of sophistication, you don't realize how completely overwhelming it is," he said. "It's a company that had a colossal, concerted, sophisticated actor after them, and dealing with it and the implications is very, very painful."

When asked Tuesday about backdoor DLL code still being present on SolarWinds products as of Monday, a SolarWinds spokesperson pointed SearchSecurity to an excerpt from the 8K filed with the U.S. Securities and Exchange Commission (SEC) Monday.

"Based on its investigation to date, SolarWinds has evidence that the vulnerability was inserted within the Orion products and existed in updates released between March and June 2020 (the 'Relevant Period'), was introduced as a result of a compromise of the Orion software build system and was not present in the source code repository of the Orion products. SolarWinds has taken steps to remediate the compromise of the Orion software build system and is investigating what additional steps, if any, should be taken. SolarWinds is not currently aware that this vulnerability exists in any of its other products," the filing read.

The filing added that the vendor currently believes that "previously affected versions of the Orion products that were updated with a build released after the Relevant Period no longer contained the vulnerability; however, the server on which the affected Orion products ran may have been compromised during the period in which the vulnerability existed."

UPDATE 11/18: In a blog post, Microsoft's Threat Intelligence Center said it discovered a second malicious DLL file with backdoor capabilities during investigations of compromises. "In an interesting turn of events, the investigation of the whole SolarWinds compromise led to the discovery of an additional malware that also affects the SolarWinds Orion product but has been determined to be likely unrelated to this compromise and used by a different threat actor," the blog post said.

Regarding the aforementioned support page, which is currently live and last updated on Nov. 10, it explicitly says to exclude Orion product folders from antivirus or antimalware scanning.

"For SolarWinds products, to prevent possible application related issues, unexpected behavior and performance related problems, at minimum you would need to consider excluding the following items from antivirus or security software that you install on your SolarWinds Primary, Additional, HA backup polling engines and any web servers that you run," it read.

The recommendation has come under fire from cybersecurity experts such as Costin Raiu, director of Kaspersky Lab's global research and analysis team. "This is nuts," Raiu said in a Tweet.

However, the page has a disclaimer at the bottom noting that the information on the page "may come from third parties."

"Disclaimer: Please note, any content posted herein is provided as a suggestion or recommendation to you for your internal use. This is not part of the SolarWinds software or documentation that you purchased from SolarWinds, and the information set forth herein may come from third parties. Your organization should internally review and assess to what extent, if any, such custom scripts or recommendations will be incorporated into your environment. You elect to use third-party content at your own risk, and you will be solely responsible for the incorporation of the same if any," it read.

SearchSecurity asked SolarWinds for more context on the advisory, but the company did not offer comment at press time.