Experts: Disaster recovery plans may overlook major outages
Disaster recovery needs to provide protection from natural disasters such as hurricanes, fires and floods, as well as cyberattacks that concern many IT pros.
Data protection experts say that most businesses have their heads in the sand when it comes to a disaster recovery plan that is fully prepared to handle a hurricane, flood or fire.
While ransomware attacks are frequently in the news and businesses are taking them seriously, data center-threatening disasters are not always top-of-mind issues for IT pros.
Although many businesses use the cloud either for backup or for failover during outages, or use disaster recovery as a service (DRaaS) to help ensure business uptime, it’s important to have an all-encompassing disaster recovery plan that offers protection from a major natural disaster. Blind spots include not preparing for lack of internet, power or physical access to the data center after a disaster.
Marc Staimer, president of Dragon Slayer Consulting, said recovering from ransomware and natural disasters pose a different set of challenges. A ransomware victim is in danger of restoring from compromised backups or losing access to the backups entirely. Natural disasters don't specifically aim for an organization's backup data, but they can do far more damage to infrastructure. Ideally, businesses are prepared for both.
Steven Hill, senior analyst at 451 Research, said DR must go beyond simply failing over critical applications to the cloud. However, many businesses do not take them because of the extra cost and the belief that they will never suffer a major outage. He likened business continuity/disaster recovery (BC/DR) to an insurance policy, and not everyone is willing to pay more for comprehensive coverage.
"BC/DR is like buying insurance," Hill said. "It's a matter of how much do I want to pay."
Hill said the cloud is a good insurance policy for business interruptions that don’t affect the data center physically. These interruptions come from ransomware and other cyberattacks, which can hit anywhere and at any time. But he added that these types of business interruptions are not "disasters" in the traditional sense.
A true disaster is something that physically impacts the data center and renders it inoperable, either by destroying hardware, the building it's housed in or the personnel who work in it. George Crump, president of IT analyst firm Storage Switzerland said most businesses haven't suffered -- nor are they prepared for -- a full-on disaster.
"Assume your data center's gone. Now what?" Crump said.
Crump found from his experience that most organizations’ disaster recovery plans are really just best efforts, with no formal plan in place. What they do have has worked for them in the past, so they don't invest in anything more structured. Crump compared it to how people who have never had their homes broken into aren't likely to install a home security system.
"Most people have a casual or cavalier attitude toward DR," Crump said. "It's not a lack of education or lack of access to education."
Staying safe in the cloud
Successful disaster recovery plans often include the cloud, either one of the public cloud giants or local managed service providers.
KCF Technologies, an industrial diagnostics company based in State College, Pa., chose the cloud for its core business software as a way to protect the overall business against natural disasters. Its diagnostic software is hosted in a Cassandra database on Amazon Web Services (AWS), it uses Office 365 for email and most of its business tools such as the phone system and accounting are hosted externally.
Myron Semack, chief infrastructure architect at KCF, said building a robust data center with redundant everything, then doing the same at a separate DR location, was too cost-prohibitive. With most of the business in the cloud, KCF can shift employees around the company's multiple offices or have them work from anywhere with internet access during a disaster scenario.
"We made a conscious decision to rely on cloud-hosted SaaS tools as much as possible so we could be resilient," Semack said.
Semack said he is confident in KCF's ability to function during a natural disaster. The company's core SmartDiagnostics software is designed to work even if one or more AWS data centers go down.
Instead, Semack is more worried about ransomware attacks. Natural disasters can be prepared for, as they tend to give plenty of notice. Semack said this is never the case when it comes to cybersecurity, and all he can do is look at and adopt new security measures while keeping on top of new threats as they emerge.
"Security is never a 'solved' problem," Semack said.
‘We want to touch our data’
Ron Sinopoli, CIO of James McHugh Construction Co., said the Chicago-based firm set up its DR to guard against ransomware, but it will also enable recovery from natural disasters or other network disruptions.
“We’ve put an exorbitant amount of time and effort into security to make sure that doesn’t happen,” Sinpoli said of a successful ransomware attack.
Sinopoli said for McHugh, setting up its DR was a four-step process that includes on-premises devices plus DR as a service from a local MSP. “First, you have to pick the right product,” he said. “Second, let’s understand what we have to replicate and how often. Third, we make sure we replicate off-prem. Now we’re in the final phase -- do we want to push everything to the cloud and use Google, Amazon or Microsoft Azure -- or do we want to go locally? It’s important for us to be able to touch our data, feel our day, that’s why we decided to go with a local provider to set up a private cloud for us.”
McHugh overhauled its on-premises storage in 2018 with a new Pure Storage FlashArray SAN and Rubrik data protection appliances. One Rubrik device is on premises and the other is at a colocation site, and McHugh replicates data between them. When McHugh contracts with an MSP, Sinopoli said it will move the second appliance to the provider’s site.
Very few organizations have a good DR plan. Very few humans can really imagine what they have to do in an outage. We tend to underplan.
Marc StaimerFounder and president, Dragon Slayer Consulting
“If we lose our network at any time, whether it’s from a power outage, loss of ISP, fire or flood, we have a protocol in place in terms of how long we wait before we start to point everybody to the new data location,” he said. “You don’t want to wait too long, because it’s not easy to bring the data back into your primary environment. That can be painful.”
Sinopoli said he won’t rule out ever using a public cloud, but he said he likes to know exactly where his data is stored.
“We are still in the process of thinking through our journey to the cloud,” he said. “We want to be able to put our hands on the appliance that had our data. If the building caught fire or the server room flooded, we can pick up the phone and say ‘Hey, we’re coming to get our appliance because we need it’ instead of calling a cloud provider and not knowing who you’re going to get. Maybe in five or six years, we’ll feel comfortable not having an appliance we can touch and feel. But right now we need that extra security.”
Data growth is making recovery harder
Not all businesses are well-prepared, however. Data protection and management vendor StorageCraft recently commissioned a study that found only 15% of its respondents, all small and mid-size businesses (SMBs), could recover within an hour from a severe data loss event. About 25% said it would take between one day and one week to recover. The survey of 709 IT decision makers for companies ranging from 100 to 2,500 employees concluded that unchecked data growth was leading to longer recovery times while also opening up new vectors for cyberattacks.
Dragon Slayer’s Staimer said thanks to the cloud, the barrier of entry to a basic disaster recovery plan is low, so many businesses are prepared for small outages. However, he stressed there is a difference between prepared and "fully prepared." Because the cost and complexity for the latter is much higher, most organizations don't pursue it until after they've suffered a loss.
"Very few organizations have a good DR plan," Staimer said. "Very few humans can really imagine what they have to do in an outage. We tend to underplan."
Staimer said places where that level of readiness is a requirement, such as banks and hospitals, do tend to have solid disaster recovery plans in place. But most businesses treat DR like insurance, echoing Hill's sentiment that they only buy in to the level they feel they need.
