SolarWinds response team recounts early days of attack

The days immediately following the discovery of SolarWinds breach were chaotic, confusing and marked by more than a few strokes of good luck.

During an RSA Conference webcast Thursday, the group of executives from CrowdStrike, KPMG, DLA Piper and, of course, SolarWinds spoke about the immediate response to the 2020 supply chain heist on the massive IT software vendor. The panelists were some of the first people on the scene of what would become a historic series of attacks that affected tens of thousands of companies.

Security vendor FireEye was first to discover that nation-state threat actors had breached its network and accessed sensitive information, including red teaming tools. The company disclosed the breach on Dec. 9 and later traced to breach to a backdoor within SolarWinds' Orion software.

The first word of the attack landed over that weekend when FireEye informed SolarWinds of its finding on Dec. 12. Timothy Brown, SolarWinds' vice president of security, said initial hours progressed rapidly as FireEye provided SolarWinds with the initial evidence of an attack, and the vendor quickly confirmed that copies of its Orion monitoring software had been seeded with malicious code that allowed the attackers to spy on users.

"We didn't have to do a lot of side research to determine that this happened," Brown said. "Then we did some analysis to determine when it happened, and we figured out that three builds were affected. At that point, we knew, since something happened internally, that we really had to get the right people together for an investigation."

Ronald Plesco, partner at law firm DLA Piper, served as the breach coach for SolarWinds and had to assemble a team of incident response specialists quickly. In addition to the pressure to respond quickly and identify the source of the breach, he said the incident response team also had another "shot clock" running because FireEye wanted to publicly disclose their findings that Sunday. "Getting all of those teams in place, dividing up the division of labor and who's going to do what was key in the early days from a project management standpoint," Plesco said.

After bringing in CrowdStrike and KPMG, the incident response team sought to determine how the backdoor, dubbed "Sunburst," ended up in Orion. One of the first culprits in the attack, according to David Cowen, KPMG's managing director, was an oddly placed virtual machine (VM). After poring over the development and orchestration servers, it was an eagle-eyed developer who noted something odd within the VM.

"As we kept going farther down the stack, one of the developers said we found this powered-off VM, and we found on there the compiled bad code," Cowen said.

Even after uncovering the culprit, getting a grip on the inserted code and who put it there was not easy.

"There were a lot of loops and tricks that the adversary threw in there to make it difficult to comprehend," said Adam Meyers, senior vice president of intelligence at CrowdStrike.

One of the tricks the attackers employed was to simply cover up their native language. The way investigators are often able to attribute an attack is to analyze the simple languages cues in code, such as making comment references in Cyrillic or Chinese, for example.

In this case, however, the SolarWinds intruders were leaving no such clues. The investigators found the source code inserted into Orion had been scrubbed of any localization that would have given away the author's identity.

"That effectively laundered the code," Meyers noted. "That is when it hit me that this was next-level in terms of operational security."

In the end, investigators pored over some 100 TB of data in order to produce their report on the attack.

"It was so many moving parts and so many pieces," Brown noted. "Realizing that you needed to have teams specialized in the right spots, performing actions that were independent but related. It is controlled chaos, but with the right people involved, you can push through it."