Public sector IT execs detail ransomware backup, recovery tips
City governments are particularly vulnerable to ransomware. These city IT leaders discuss how organizations can improve data protection to be ready for the next attack.
On the same day IT director Darryl Polk recorded a session for VeeamON that focused on ransomware backup and recovery strategies, his city experienced -- what else -- a potential ransomware attack.
"That's a way to energize your morning," said Polk, the chief innovation officer for Rancho Cucamonga, Calif. "This is when having good backups is a real nice thing."
Thankfully, the attack, which included a message that claimed human rights violations by the U.S. government, was contained to an edge switch and was not successful, Polk said at last week's virtual VeeamON user conference session "Disaster Recovery Best Practices in the Public Sector."
A ransomware backup and recovery strategy is critical for both the public and private sector. In just the last few weeks, there have been several high-profile ransomware attacks, including the Colonial Pipeline hack and an attack on backup vendor ExaGrid. In both of those cases, the organizations reportedly paid multimillion-dollar ransoms.
A layered approach to ransomware backup and recovery
Sarasota, Fla., was hit with ransomware in February 2016, Director of IT Herminio Rodriguez said in the VeeamON session. The attack started with a phishing email when a user gave out an ID and password.
About 12 TB of data was gone. HR was down and the financial system wouldn't start.
"That's a really scary position to be in," Rodriguez said.
At left, Jeff Reichard, senior director of enterprise strategy at Veeam, moderated a panel of IT executives. Then, from left to right: Herminio Rodriguez of Sarasota, Fla.; Garrett Griswold of Geneseo, Ill.; and Darryl Polk of Rancho Cucamonga, Calif., spoke about ransomware backup and recovery best practices.
IT spun up Veeam, which protects all of its services, and did a full restore. The city was only down for about 15 business hours and did not pay any ransom.
"At no point were we nervous about our backups," Rodriguez said.
"With ransomware, it's not a matter of if. It's when," Rodriguez said.
In the last year, the COVID-19 pandemic has put IT on even higher alert. With the shift to remote work, cybersecurity defenses across the world weakened.
"It seemed like there was just a massive surge in malicious actors taking advantage of the crisis," Polk said, but his city's Veeam backup system gave him more confidence.
Another new trend is an attack that permeates the network but doesn't execute immediately. The attack inspects the network and goes after backups.
We want to get backups away from the bad guys.
Herminio RodriguezDirector of IT, city of Sarasota, Fla.
"We want to get backups away from the bad guys," Rodriguez said.
A ransomware backup strategy should include an offline component. For example, the city of Sarasota uses tapes for one piece of its backup. The city stores them about an hour away in a hardened building -- the tapes aren't connected so they can't get hacked.
"That helps me sleep at night," Rodriguez said.
In a similar way, the city of Geneseo, Ill., a community of 6,500 residents and a customer of Veeam and Iland for backup and recovery, is careful about where it stores certain data. For example, body camera data from the city's 13 police officers does not go to the cloud, said Garrett Griswold, director of IT. Rather, that data goes to a cold site, he said in the VeeamON session.
Plan and test to improve confidence
Testing is a critical but often overlooked piece of not just a ransomware backup strategy, but also good data protection hygiene, in general.
Rancho Cucamonga, a southern California city of 177,000 residents, recently held an executive-level tabletop exercise of what a data breach would look like. It simulated a ransomware attack.
"What I wanted them to do is feel the moments before we know that we're going to be able to restore and what that initial response is going to be," Polk said.
Sarasota, which normally has 55,000 residents but increases in population to 120,000 in the winter, consistently tests its backups.
"That's something that a lot of people don't do today," Rodriguez said.
"That's a great document where it's all laid out," and IT can reach for it at a moment's notice, Rodriguez said. It's important for the business side to be familiar with the plan.
"Cybersecurity is definitely a layered approach," Polk said. "This is not a one and done."
