At RSA Conference, Colorado CISO Deborah Blyth gave an inside look at the state's response and recovery effort following a devastating SamSam ransomware infection in 2018.
SAN FRANCISCO -- The SamSam ransomware attack on Colorado's Department of Transportation ultimately made the state's cybersecurity program stronger -- so much so that it likely prevented repeat attacks.
At RSA Conference 2020 on Monday, Colorado CISO Deborah Blyth gave a behind-the-scenes look at the highly publicized SamSam attack that crippled the Colorado Department of Transportation (CDOT) in February 2018. Despite the incident, Blyth said her team learned valuable lessons that benefited the state government's security program.
"Two years later, our program is so much stronger as a result of the ransomware attack," she said during a seminar on emerging threats.
The vector in the SamSam ransomware attack was a misconfigured virtual server that threat actors used to access the CDOT network. The system was intended to be a short-term testing server, but standard security controls weren't applied -- and within 48 hours, the server was compromised.
"The system was under attack the day it was created," Blyth said. "And after 40,000 password attempts, the system was breached."
Threat actors used the virtual server to access the rest of CDOT's network and used the domain controller to push out the SamSam ransomware. Luckily, Blyth said, the ransomware was limited to CDOT's business operations; CDOT's network segmentation limited threat actors' ability to move laterally and spread the ransomware to critical areas like traffic control systems.
Colorado CISO Deborah Blyth discussed the 2018 SamSam ransomware attack onthe state's Department of Transportation.
Still, the back-office systems included nearly half of CDOT's entire computing environment; approximately 1,300 workstations and 400 servers were instantly encrypted, Blyth said. And while the data was backed up, the state's security team was tasked with removing the threat and restoring service.
Calling the National Guard
Blyth said the initial incident response effort was hampered by setbacks. A week after the attack, Colorado's response team members thought they were clean and began bringing systems back online -- only to discover that night that there was new threat activity in the network and more instances of SamSam. "It was clear at that point that we did not have containment," she said.
In addition, Blyth said after the attack the security team was investigating "every blip and anomaly" on the network, which made it hard to effectively respond to the SamSam incident. "We called this 'chasing ghosts,'" she said.
As a result, Colorado was forced to declare the first ever state of emergency for a cyber incident. Blyth said the declaration was important to the recovery effort because it allowed her to coordinate with the state's Office of Emergency Management, which in turn granted her team access to the Colorado National Guard.
"What most people don't know is that most states have what's called a 'defense cyber operations' element within their National Guard," she said.
Blyth's response team worked with infosec experts from the Colorado National Guard, meeting with them twice a day. Along with the Office of Emergency, Blyth's response team was aided by Colorado's state fusion center, the Governor's Office of Information Technology, US-CERT, the Department of Homeland Security, the Federal Emergency Management Agency and the FBI, along with four unnamed security vendors and an incident response firm.
The FBI was on site "the entire time," investigating the attack and collecting evidence. Less than nine months after the CDOT incident, the Department of Justice indicted two suspects connected to the SamSam attacks. "We were thrilled," Blyth said. "We felt like we hopefully contributed to that as well, in providing all of that evidence to the FBI."
More than 130 people were involved on site in the response and recovery effort, many of them working 18-hour days. Two weeks were spent eliminating all threat activity and removing the malware, and another two weeks were spent on restoring systems and services, which cost a total of $1.7 million.
Lessons learned
Blyth said the response effort gave the state government an opportunity to improve its infosec posture and address weaknesses across the entire environment. Some needs included more training for cloud security and governance, better controls for privileged accounts, and increased visibility into all the state's computing resources and assets.
In addition, the team took several steps to improve the statewide security posture, including closing instances of Microsoft's remote desktop protocol that were open to the internet, and rolling out two-factor authentication for all privileged accounts.
One of the most important lessons involved timing. Blyth said the state had invested in a new, unnamed endpoint detection and response product and was in the process of deploying it. Unfortunately for the CDOT, the product rollout was scheduled for the week after the SamSam ransomware attack occurred.
One of the things we realized was that we had all of the right security improvements and initiatives already under way. We just needed to be faster at executing them.
Deborah BlythCISO, State of Colorado
"One of the things we realized was that we had all of the right security improvements and initiatives already under way. We just needed to be faster at executing them," she said.
Blyth said a few months ago, the security operations team detected several instances of Trickbot malware in the network. Luckily, the team had new products in place that identified Trickbot and prevented it from executing. "We think this could have been another event on par with the SamSam incident, but we were able to cut it off at the pass," she said.
It's possible a repeat attack could have been worse for Colorado, since some threat groups have combined data exposure with their ransomware attacks. Matt Olney, director of threat intelligence at Cisco Talos, told SearchSecurity earlier this month that his team had seen recent ransomware attacks start with a Trickbot or Emotet infection that gives threat actors initial access to a network. The threat actors use the access to exfiltrate sensitive data before encrypting systems, and then threaten victims with the public exposure of that data.
In the end, Blyth said the state made the right decision to not pay the ransom, bring in both state and federal partners for response and recovery, and to act on the lessons learned from the SamSam attack.
"While the impact was certainly felt by CDOT employees, it was isolated to a single network, we recovered 100% of all product data without paying a ransom, and it was at a fraction of the cost of what you've heard other well-publicized events costing," she said. "The state of Colorado really does consider the CDOT response and a recovery a success."
