MSP shares details of Kaseya VSA ransomware attack, recovery

On July 2, 2021, Progressive Computing Inc., an MSP in Yonkers, N.Y., was the victim of a supply chain ransomware attack. REvil threat actors exploited a zero-day vulnerability in Progressive's remote monitoring and management tool, Kaseya VSA, affecting all of its 80 customers and 2,000-plus endpoints

In the end, Progressive was one of 60 Kaseya MSP customers affected.

Now, Robert Cioffi, co-founder and CTO of Progressive, is looking back on his experience and sharing the process of discovery and recovery. One of the messages he wants to hammer home is just how critical of a role community, which includes Axcient and its X360Recover platform, played in getting the 30-year-old MSP back on its feet.

"As an MSP, you're not designed to support all of your customers 100% of the time. It's fractional services. Now, 100% of my customers needed us 100%," Cioffi said. "I didn't have enough bodies. Nobody does -- that is the way we're built, by design. We had a lot of help from peers and vendors. It was community that saved our asses."

Editor's note: This Q&A has been edited for clarity and conciseness.

Describe the attack.

Robert Cioffi

Robert Cioffi: It was a classic smash and grab -- meaning it was a lightning strike through the supply chain, and it was intended to push ransomware out to as many endpoints as possible and as quickly as possible. So there was no targeted data exfiltration. There was no lingering around for weeks pulling data out or attacking backup systems and deleting backups.

How were you alerted to the fact that you were a victim of ransomware?

Cioffi: Our phones rang off the hook -- that was the first thing because the attackers used our RMM [remote monitoring and management] tool, Kaseya, to push ransomware. They broke through all the layers of security and took advantage of a zero-day exploit within the RMM platform and gained administrative control of the system.

You called it a lightning strike.

Cioffi: Yes, the speed at which they operated -- they didn't even bother to lock us out. They just set up an admin account, uploaded their ransomware and uploaded a script to push ransomware to every single endpoint we manage.

How did this kind of attack affect your recovery?

Cioffi: Because of the speed of the attack and the opportunity window that was open, their target was a very wide net. If they were specifically targeting my company or one of my customers, they could have done a lot more damage with the additional time on their hands, but they had no time for anything. Thankfully, in this case, the backups were not compromised.

With recovery, it was a matter of [understanding that] we had to restore every server that was 'ransomwared' to a safe recovery point. We saw one of mainly two scenarios -- either there was a local appliance holding snapshots of backup data or customers were in a virtualized model where backups were stored in the Axcient cloud. Actually, all images are stored in the Axcient cloud -- the only difference is whether there's a local copy.

Is there a benefit to having a local copy?

Cioffi: Speed of recovery -- because if I have a terabyte of data to pull down from the cloud to your office, we're going to be waiting.

When did you reach out to Axcient?

Cioffi: That afternoon. The attack started at 10:49 a.m. ET with the attack on our Kaseya VSA instance. It wasn't until about 12:15-ish that we started getting calls from clients, and then it wasn't until about 12:30 that we were fully aware of what had happened.

We didn't know the how, we didn't know the why, but we knew the what. And the what was that every single one of our customers was ransomwared and they used our Kaseya VSA instance to carry out the attack. That was all we knew. At that point, we started contacting all of our various vendors, our security vendors, as well as Axcient, to say, 'Hey, we've got this situation, and we're going to need some help.'

How did Axcient help with recovery?

Cioffi: They provided us some technical support to augment our team in our recovery efforts. They don't actually do the recovery, but in our time of need and pain, they were a true partner. They gave us resources such as senior sales engineers who flew out to New York and helped us with some recoveries, as well as being on the phone with us to support our high-level engineers to assist with any recovery roadblocks.

Did you run into any roadblocks?

Cioffi: We ran into some issues, but they were things like hardware failure or storage space hitting odd limits. These were physical kinds of barriers that got in the way -- we had to free up some space and restart that recovery because we ran out of disk space or we needed to throw some more drives in there to make enough room so we could carry out the recovery. They were helpful and supportive in that way. And thankfully, Axcient's technology worked.

Tell me more about the hardware failure. What happened?

Cioffi: The production server we were trying to restore to -- and I'm laughing about it now, but it was a horrific day because it was one of the first systems that we tried to restore -- sure enough, right out of the gate, the production server hardware failed.

We ran into a couple little issues like that, but for the most part, it was relatively smooth sailing. The next biggest thing was lack of storage, because you're trying to keep multiple copies of things and restore to production servers that may be a bit older, and you've got to worry about storage limitations. We ran into a couple of problems like that -- nothing really all that showstopping.

How long did it take to completely recover?

Cioffi: Eighty customers were restored in 17 calendar days. And that's including the wait time of the Fourth of July weekend when we were stalled by insurance and legal and forensics. ... We needed to understand the nature of the attack before we could formulate a remediation plan and then begin figuring out how to execute on that remediation plan.

You expect as an MSP to have a customer who is going through a security incident. You don't expect 100% of your customers to go through a full-scale attack simultaneously.

How much of this plan did you have to formulate after the attack versus what you had in place as part of your DR strategy?

Cioffi: Anyone who says, 'Oh, we've got a plan for that' -- BS. You don't, because no one expects to go through what we did. You expect as an MSP to have a customer who is going through a security incident. You don't expect 100% of your customers to go through a full-scale attack simultaneously. And even if they say that they've got a plan, I doubt they would have the resources in place to be able to pull the trigger -- I don't care how big that organization is. ...

I don't want to make it sound like I don't bother planning. I think you can plan for it. It's just that it's so absurd to think about an RMM attack like that. Even though lots of people worry about it, no one's really put together a plan for that.

Why is that?

Cioffi: I think part of the reason is because, No. 1, anyone who has been attacked like me has not had -- and I'm going to use the word courage -- has not had the courage to get up in front of crowds and openly talk about it. I have exposed myself to the world through storytelling because I firmly believe that there is no shame in victimhood and that there is strength in vulnerability. How do I help the community rise up to the next event if I don't get up there and talk about things that we went through?

No. 2, you have to have a strong community. That's one of the missing elements in the cybersecurity conversation.

Why was that community so crucial?

Cioffi: The ransomware event did not attack any cloud-based system because it was meant to install ransomware on Windows-based devices. Data that's stored in Teams or some line-of-business application -- think QuickBooks Online -- those systems were unaffected. Microsoft 365 email -- unaffected.

It was the machines that were affected. I had to restore servers to get all that data back for our customers, and there were 250 servers that we needed to recover. That's a mountain of work. And then every single endpoint -- over 2,000 computers, desktops, laptops -- were completely ransomwared. We had to send a technician to sit at your desk and reinstall Windows and put all your settings back and get you reconnected.

That was the bulk of the work. I had a team dedicated toward restoring servers, and then I had a team dedicated toward restoring desktops and laptops. The community really helped with the latter part. That has nothing to do with storage, but the assistance that we received ... [gave] us the manpower to dedicate to those tasks so that I could focus my attention on getting servers restored.

Nicole Laskowski is a senior news director for TechTarget Editorial. She drives coverage for news around enterprise applications, application development and storage.