SolarWinds breach highlights dangers of supply chain attacks

The extreme dangers of supply chain cyber attacks are once again on display following the hack of SolarWinds by suspected nation-state threats actors.

While such attacks have occurred previously and on smaller scales, security experts say this one was catastrophic. FireEye, one of SolarWinds' 300,000 customers, last week disclosed it had been breached and its red team tools were compromised. On Sunday, SolarWinds confirmed it was the victim of a supply chain attack conducted by nation-state hackers. The threat actors planted a backdoor in software updates for SolarWinds' Orion platform, which were issued to customers such as FireEye and various U.S. government agencies.

Gartner research vice president Peter Firstbrook told SearchSecurity that the breach is significant due to the scale of the potential victims and the access threat actors were able to gain in terms of legitimate IT utilities, which may have had excessive permissions. Specifically, Firstbrook said the attackers may have used an technique called Kerberoasting, where adversaries exploit a weakness in the Kerberos authentication protocol to crack passwords.

"In this case the attacker was able to get a certificate and sign a fake DLL helper which was then used to get a backdoor running in the SolarWinds application and then use that to monitor the network and move laterally. It looks like Kerberoasting was the lateral movement," he said. "Endpoint detection and response (EDR) tools are critical to detecting these types of attacks and to search history. Only 30% of endpoints have EDR capabilities so the industry has a long way to go."

While the access obtained by the threat actors was devastating, Firstbrook said it appears that not all Orion customers that received the malicious updates were breached. "The incursions were reported by FireEye as manual and aimed at high value government targets. It is not a worm, so most organizations were likely not targeted," he said in an email to SearchSecurity.  "Previous supply chain attacks include the most destructive worm ever: NotPetya."

The notorious NotPetya ransomware attacks of 2017 involved the compromise of a Ukrainian accounting software called M.E.Doc, which threat actors then used to spread the ransomware to various businesses. The U.S. government in 2018 publicly attributed NotPetya to Russian state-sponsored hackers, and in October, the Department of Justice indicted six officers with the Russian Main Intelligence Directorate (GRU) in connection with the ransomware attacks.

Cybereason CEO Lior Div said there are similarities between the NotPetya supply chain attack and the recent SolarWinds compromise. Several media outlets have reported that APT29, a Russian state-sponsored hacking group also known as Cozy Bear, was behind the SolarWinds campaign. Cybereason, which was part of the team involved in the NotPetya investigation in 2017, agrees with that assessment.

"We are sharing the same belief that the SolarWinds hack was the APT29. This is not the first time we've seen the Russians using this method. We saw it in NotPetya. It's a known technique the Russians are learning," he said.

For a supply chain attack of this nature, Div said, the amount of manpower and time needed to prepare, and the accuracy required by the threat actors, make it very difficult to achieve. But he said the attack also demonstrates what's possible when threat actors gain access to a major vendor's supply chain. "When someone like APT29 decides to go after you, they will be able to go after you."

Huntress Labs, a managed detection and response vendor, investigated the SolarWinds attack on behalf of its managed service provider (MSP) clients (SolarWinds provides remote management tools used by MSPs). In an email to SearchSecurity, Kyle Hanslovan, CEO of Huntress and John Hammond, senior security researcher, said the compromise of SolarWinds' supply chain has had devastating effects.

"This supply chain attack was sophisticated and expertly executed," they said. "In a strange way, we almost have to tip our hats to the hackers. Although the scenario could have been worse -- in the hypothetical case all SolarWinds products were compromised -- this is undoubtedly a dire situation. "

Malwarebytes Labs called the SolarWinds hack possibly the largest hacking campaign of 2020. "This scenario, referred to as a supply-chain attack, is perhaps the most devious and difficult to detect as it relies on software that has already been trusted and that can be widely distributed at once," the Malwarebytes threat intelligence team wrote in a recent blog post.

Breaking supply chains

It is unclear how SolarWinds was compromised, but the company believes, according to its 8-K filing on Monday, the backdoor was "introduced as a result of a compromise of the Orion software build system and was not present in the source code repository of the Orion products."

On Monday, Volexity threat researchers Damien Cash, Matthew Meltzer, Sean Koessel, Steven Adair and Thomas Lancaster published a blog titled, "Dark Halo Leverages SolarWinds Compromise to Breach Organizations." The post provides insight into how SolarWinds was used by the same suspected threat actors in one specific breach in July to gain full control of a victim network.

"Volexity identified suspicious administrative commands and ActiveSync anomalies in the organization's Exchange environment. Further review of the organization's endpoint software and network traffic confirmed a breach. The attacker had executed commands to export e-mail for specific users in the organization, and then exfiltrated the data via the organization's Outlook Web Anywhere (OWA) server," Cash, Meltzer, Koessel, Adair and Lancaster wrote in the blog.

The researchers also describe how APT29, which Volexity tracked as "Dark Halo," breached some targets prior to the SolarWinds compromise by bypassing 2FA.

Another example of weak security posture was discovered by security researcher Vinoth Kumar. He posted to Twitter Monday that he found a public GitHub repository leaking file transfer protocol (FTP) credentials that belong to SolarWinds. Kumar told SearchSecurity he found the repo on Nov. 19, 2019 through manual GitHub reconnaissance and discovered it contained a password to a SolarWinds update server that was "SolarWinds123."

"That repo had FTP credentials and FTP server is accessible in public. Then I reported this issue to SolarWinds on 19th and the issue was fixed on 22nd Nov. The concern was credentials was very weak and the FTP server is public," he said in a message to SearchSecurity.

Responding to supply chain threats

As the nation-state attacks take aim at the supply chain, it's proving that changes are necessary to minimize security risks.

Ryan Young, CTO of integration services firm Vandis, told SearchSecurity the SolarWinds attack should change how people patch, update and certify software packages moving forward. But he also said the incident may make organizations more conservative about applying patches.

"One of the most important things the industry has done is push our customers, push ourselves to be better about having a much faster patching cycle, and I think stuff like this will make people weary of doing that," he said.

Huntress Labs' Hanslovan and Hammond agreed.

"The entire security industry, Huntress included, trusted the integrity of SolarWinds software and the reputation of their digital signatures," they said in an email. "As a result, the compromised DLL came from a seemingly trusted and legitimate update, and we all have paid the price. In 2020, all mature organizations believe in 'patching early and often,' and the attackers capitalized on that."  

Overall, Young said it will get the industry to start looking at the importance of next-generation firewalls and move beyond just layer 4 network inspection. "This will help address a hygiene issue that exists in the enterprise today and that is not enough people are decrypting the encrypted traffic to inspect it for theft of exfiltration of data."

Many organizations are responding to the SolarWinds attack by conducting in-depth reviews of their infrastructure to rule out any supply chain threats. Bill Terwilliger, CEO of Boston-based security consultancy Alpha Defense, said his incident response business has seen a noticeable uptick following the disclosures of the attacks on FireEye and SolarWinds.

Young said he had several requests from multiple customers today asking him to validate that Vandis is not running SolarWinds in the environment and to confirm there has been no exfiltration of data. When the breach first broke over the weekend, Young said his team started calling customers to find out who had Orion running and help them come up with plans to mitigate the threat either by shutting them down or taking some of the guidance provided by SolarWinds and FireEye. If possible, they recommended shutting down as fast as possible.

During Forrester's Security and Risk 2020 virtual conference on Sept. 23, analyst Alla Valente hosted a session titled, "Supply Chain Risk: You Can't take Your Eye Off the Ball." In the session, she talked about the struggles supply chains face like deciding between being more efficient or more resilient.

"It's also a product of decentralization. For example, we see there is no defined or single owner for risk management of the supply chain. So, unlike departments like financial or legal that owns legal risk, when it comes to managing risks of vendors, suppliers and other third parties they get bounced around like a hot potato," she said during the session. "When there's so many people involved in the process and there is no set of accountability, everyone's responsible and no one is accountable. In this study, we see in the last three years management and top-level support for risk is waning. And the tools that we have to predict, monitor or even measure risk or disruption in our supply chain are wholly inadequate."