MSPs scramble to bolster security amid ransomware spike
After a flurry of devastating ransomware attacks in 2019, MSPs and vendor partners are improving security to prevent history from repeating during the pandemic.
Many companies suffer data breaches, but as Tom McDonald learned last year, a breach for a managed service provider can lead to devastating consequences.
McDonald's IT support and cybersecurity company, NSI in Naugatuck, Conn., was compromised by threat actors last June. The attackers used the MSP to infect more than 20 of NSI's clients with ransomware.
"They came in through us," he said.
NSI isn't alone. Before 2019, MSPs didn't view themselves as high-value targets for cybercriminals. But all of that changed last year as a series of devastating cyberattacks like the one NSI experienced ravaged not just MSPs but their clients as well. These attacks involved threat actors who use the MSP as a launchpad to spread ransomware to clients. In some cases, the initial attack started with simple phishing emails or brute-force attacks on accounts with weak or reused passwords.
In other, sometimes more severe cases, threat actors exploited vulnerabilities in remote access and management tools popular among MSPs. For example, in February 2019 threat actors exploited a known vulnerability in a ConnectWise plugin -- which had been patched more than a year earlier -- to compromise at least four different MSPs, spreading GandCrab ransomware to their respective clients.
Kyle Hanslovan, founder and CEO of threat detection vendor Huntress Labs, said 2019 was a turning point for MSPs. By June 2019, Huntress Labs, which caters to MSPs and SMBs, saw an average of three to five MSPs compromised per week, he said.
"It was the summer of Sodinokibi [ransomware]," Hanslovan said. "We had 100 MSPs that we work with get breached and had their remote management tools used to deliver ransomware to clients, and after that we just stopped counting."
Juan Fernandez, vice president of managed IT services at ImageNet Consulting in Oklahoma City, said ransomware attacks via MSPs were so bad that they "blackened the eye of the MSP brand."
Now, during the COVID-19 pandemic when remote access has surged, MSPs and their vendors are applying the lessons learned from 2019 to prevent a repeat of history.
Warning signs for MSP security
While MSP attacks reached a boiling point last year, there were many warning signs before that. In 2017, threat researchers and law enforcement agencies revealed an extensive cyberespionage campaign from a Chinese state-sponsored group known as APT10. The group targeted large MSPs to steal sensitive data and intellectual property from their clients.
In October 2018, the Cybersecurity & Infrastructure Security Agency (CISA) issued an alert about advanced persistent threat (APT) groups attempting to infiltrate global MSPs to gain access to client networks. The CISA alert offered guidance to MSPs and their clients for detecting and mitigating such threats, including establishing and updating an incident response plan, regularly patching applications and operating systems, reviewing and monitoring privileged accounts and creating baselines for network activity.
Unfortunately, many companies did not heed the warnings.
"A lot of MSPs saw the threat as Chicken Little or the sky is falling," said Joy Beland, senior cybersecurity education director at ConnectWise, which provides IT software for MSPs. "But that all changed last year."
We had 100 MSPs that we work with get breached and had their remote management tools used to deliver ransomware to clients, and after that we just stopped counting.
Kyle HanslovanCEO, Huntress Labs
Beland, who owned and operated an MSP for more than 20 years before joining ConnectWise, said it can be a struggle for companies, especially smaller MSPs, to stay on top of all the latest threats, patches and other elements for security.
"The smaller MSPs in the SMB space don't have the resources and can't keep up with it while doing all the day-to-day stuff for their clients," she said.
ImageNet Consulting's Fernandez said when the MSP market started to take off more than a decade ago, it was a "land grab" and many companies simply wanted to sign up as many clients as they could without much consideration for security. "There was no plan for MSPs," he said. "The plan was to make money, not to be secure."
That land grab, Fernandez said, created a large threat landscape with smaller, regional MSPs that had weaker defenses and, in retrospect, were ideal targets for cybercriminals. And while the attacks in 2017 and 2018 were largely focused on national and global MSPs in cyberespionage campaigns, cybercriminals last year began to exploit those weak defenses for a different kind of threat.
Threat actors compromised several MSPs in June 2019 and infected their clients with Sodinokibi ransomware.
'A game changer'
According to a Malwarebytes report, ransomware gangs began to target MSPs in 2019 to use their remote access tools as "pivot point" to reach enterprises, a tactic that was previously used only by APTs.
NSI was one such victim compromised by cybercriminals who used the MSP's remote management connections to infect customers with ransomware.
"At the time, we had about 65 clients and a third of them were impacted by the Sodinokibi [Revil] ransomware," McDonald said. "We don't know exactly how it happened, but it was a game changer."
NSI investigated the attack and determined the threat actors gained access to the MSP's Webroot SecureAnywhere management console and used it to spread the ransomware to 22 customers. McDonald's team suspects the attackers stole console credentials from one of NSI's staff members, though it's unclear how that happened.
That incident coincided with reports of several MSP attacks in June 2019 involving Sodinokibi ransomware and Webroot. The vendor said no vulnerability was exploited in the attacks and stolen credentials were to blame. Nevertheless, Webroot updated SecureAnywhere shortly after the attacks to make two-factor authentication (2FA) mandatory for all accounts.
While NSI was able to help the majority of the 22 clients restore their data, four customers, which did not have adequate backups, ended up paying the ransom.
"It had an impact on our business," McDonald said. "We lost a lot of money and quite a few clients, and so we're kind of pulling out and recovering from that now."
While the attack damaged his business, McDonald said it was also a valuable learning experience for NSI, which responds with at least one major incident a month involving a client or related third party.
"We're well-seasoned on how to deal with these things," he said. "We went from not being able to clearly articulate security to knowing exactly what they needed and why they needed it."
After 2019, virtually every MSP knows they have a potential target on their back, but many are unsure of the steps that need to be taken to prevent breaches and ransomware attacks. For NSI, those steps include everything from implementing multifactor authentication across the board and developing an incident response plan to working with third-party vendors like SentinelOne for an outsourced security operations center.
Stories like NSI's compelled other MSPs to take action to tighten security. Penny Belluz, director of operations at Teleco in Thunder Bay, Ont., said the looming threats to MSPs forced her company to update its own operations. That included shutting down a customer portal where third parties could create their own tickets and get updates because the system presented too much of a risk.
"We're very worried about being more of a target," Belluz said. "If we tell customers to do all these things for security, then we have to do them first."
"We're not experts in how this works," he said. "You have to have partners that are 100% focused on security."
Training days
The flurry of ransomware attacks in 2019 spurred several MSP-centric vendors to push out security training, education and awareness about the looming threat. Huntress Labs, for example, has recommended basic steps like implementing 2FA for all MSP employees and using Microsoft's Group Policy for Active Directory to create additional controls for accounts.
"We put out as much education as we could telling MSPs to consider their attack surface because everything they do is attack surface," Hanslovan said.
But this year, the situation became even more complicated for MSPs with the onset of the COVID-19 pandemic. Huntress Labs saw a contraction of remote desktop connections toward the end of 2019 as its MSP customer tried to reduce their attack surface, Hanslovan said.
"We have about a half million computers under our management. Back in December, only 30,000 had external IP addresses," Hanslovan said. "But then the COVID-19 pandemic happened and working from home surged; that number shot up to about 100,000. And, unfortunately, remote desktop is being opened up left, right and all over the place."
The pandemic has had a positive side effect as well, according to ConnectWise's Beland. The vendor has offered a number of virtual bootcamps and training and certification events in recent months, which offer a more convenient and less expensive alternative to traveling to live events.
"It's the perfect time to do this," she said. "Anything we can do to bring more training and certification events to MSPs during this time, we're going to do it."
For example, a recent ConnectWise Certify training and certification event on security fundamentals for MSPs' sales teams and owners had higher than normal attendance -- 183 registered attendees, 162 of which passed the sales certification exam at the end of the day-long event.
In addition to education on security best practices, the event also offered recommendations for MSPs on strengthening their own security postures. The NIST's new guidance, "Improving Cybersecurity of Managed Service Providers," includes specific advice on addressing ransomware threats with asset monitoring and backup practices.
Brian Beck, Indiana branch sales manager at Commonwealth Technology in Lexington, Ky., said the training event was extremely valuable because unlike similar virtual events he's attended, ConnectWise Certify focused more on security practices and strategies than it did on the software vendor's own products.
He also said the event couldn't have come at a better time.
"[MSP customers] don't realize the exposure they have because of home office networks, which aren't nearly up to snuff compared to their corporate infrastructure when employees were in the office," Beck said. "They think because they're connected through VPNs that they're protected, but they're not. And if MSPs aren't having these discussions now, [the customers] are never going to know until they get taken out."
McDonald has leaned on training and education from vendors like ConnectWise to improve NSI's security posture and to help customers. He also participates in industry peer groups and has shared his experiences with other MSPs.
But he said more needs to be done to inform MSPs of ransomware threats and what needs to be done to mitigate them. He likened the situation to oxygen masks on airplanes -- MSPs, he said, need to apply their masks first before they put on their clients' masks.
"I don't think anybody really gets the impact it can have until it happens," McDonald said of ransomware attacks. "We need to be doing more to protect ourselves."
