County's ransomware remediation includes backup, security

How Yuba County recovered

Sixteen people work in Yuba County's centralized IT department. They manage services for all departments of the county, which is about 40 miles north of Sacramento and home to 79,000 residents. The most critical services are connected with public safety, including the sheriff's office, emergency services and the health department, which was especially important during the pandemic.

While some organizations would rather not admit to getting hit with ransomware, LaValley said he hopes that by telling his story, he can help businesses prevent or survive future attacks.

The infection started a few weeks earlier, in January, getting in through a PC in the building department. LaValley said it was most likely file-based, not email -- the building department deals with lots of other parties and file exchanges.

The day before that late-night phone call, the county's Active Directory experienced Kerberos issues. LaValley said they didn't realize at the time that it was part of the attack. In addition around that time, Sophos identified a Dridex infection.

The attackers then created a fraudulent enterprise admin account and encrypted 50 PCs and 100 servers.

IT staffers began the ransomware remediation process by immediately disconnecting all servers, which were about 95% virtual, so it was relatively easy, LaValley said. As a starting point for recovery, IT initially restored Active Directory from a two-week-old backup.

The department disabled admin accounts, notified businesses and users, informed authorities including the FBI, and blocked all network traffic outside the U.S. During the recovery process, it also uploaded evidence for forensic analysis.

One hundred percent of what we had backed up we were able to recover.

Yuba County had an on-premises Rubrik appliance and replicates to the cloud for disaster recovery. Within a day, it had the critical systems back -- about 20% of its total. The next group of systems took a week and the remainder took another couple of weeks. LaValley said the VMware infrastructure and lack of documentation held them up.

"Rubrik was never a bottleneck in that process," he said. "One hundred percent of what we had on Rubrik, we were able to recover."

The county performed live mount and re-imaged affected PCs and physical servers. A couple of the servers were not in the backup cycles, but LaValley acknowledged that was the county's fault and proper documentation would have helped.

As one of the last pieces of the ransomware remediation process, IT had to rebuild trust with agencies and partners to reconnect.

"It took a while to convince them we had things locked down," LaValley said.

Guidelines for ransomware remediation

Ransomware has made headlines for several years, but it was really thrust into the spotlight again this month with the Colonial Pipeline attack. In that case, Colonial Pipeline reportedly paid a $5 million ransom.

Organizations often have to pay even if they have a backup platform, said Dan Rogers, president of Rubrik. For example, Colonial Pipeline reportedly restored some of its data from backups.

"Cyberattackers are getting a lot smarter," Rogers said at Rubrik Forward, which was virtual again this year.

Ransomware that attacks backups is one recent trend. Immutable backup that attackers can't alter or delete is one way to alleviate that threat.

LaValley had several tips for ransomware remediation and prevention:

• Verify recoverable, isolated backups that attackers can't access.
• Have complete backup and recovery documentation.
• Test recovery plans.
• Understand that Active Directory, while a critical infrastructure, does not have modern security capabilities, primarily multi-factor authentication.
• Get cyber-liability insurance and understand the policy and process.
• Tighten endpoint protection.

In addition, LaValley said Yuba County is stepping up user security awareness training and phish testing.