How to plan for the worst possible disaster recovery scenarios
Your worst-case DR scenario today might be vastly different than it was just a few years ago. What's the worst that could happen to your data center in the event of a disaster?
Even as corporate data resources have grown in scope and become ever more critical, planners have orbited around a fairly orthodox view of worst-case disaster recovery scenarios. Now, recent major natural disasters in areas with lots of tech infrastructure, such as the flooding in Texas in 2018, are giving some planners pause.
What's more, with global trade and diplomatic tensions higher than they have been since the 1980s -- including a sputtering conflict between two nuclear-armed South Asian nations -- is it time to rethink DR?
Below, IT experts weigh in on some of the worst possible disaster recovery scenarios today and what can be done in the DR planning process to mitigate such crises.
What's the worst that could happen?
As Enterprise Strategy Group senior analyst Christophe Bertrand points out, worst is a relative term. While some organizations might plan for the end of the world as a worst-case scenario, others might find such planning to be overkill.
Certainly, there are examples that sound like worst-case scenarios. Consider the case of the VFEmail hack that took place earlier this year. All of VFEmail's U.S. business data was effectively deleted by hackers overnight with no warning. The company was ultimately able to recover its European operations, but the U.S. business no longer exists.
When assessing the risk of potential disasters, people typically think about three things, according to Joseph George, vice president of product management for global recovery services at Sungard Availability Services. Those three things are:
What is the probability of that scenario happening?
What will be the effect or cost if that scenario does occur?
It's critical that you consider all three components at the same time, George said, and thinking through all of those elements requires diligence.
"Most people have a natural optimism bias when it comes to disasters and don't typically believe a disaster will happen to them," he said.
If people don't think a disaster is likely to happen to them, they likely won't take action to prepare for that scenario. However, the sheer frequency and types of disasters that are taking place today are forcing organizations to change their thinking.
For Kyle Young, director of IT at security and privacy compliance assessor Schellman & Company, working for a company based in Florida presents a set of weather-related challenges that he must plan for.
"Working for an organization that is headquartered in a region that is meteorologically under siege for nearly six months out of the year -- Tampa -- planning for the worst-case scenario is an absolute must," Young said.
Of course, the most important factor for the successful execution of a business continuity and disaster recovery (BC/DR) plan is the safety of the employees, he said. Furthermore, ensuring the proper employees and teams are involved in developing, testing and executing these plans when necessary is critical to the operability of the company during an event.
According to Young, you should think strategically when selecting vendors for your BC/DR plans, preferably those that operate in multiple geographic zones to allow for continuous availability in these types of situations. It is critical for your business to have the ability to operate during a multitude of disasters that could affect your business, including meteorological events, human error or cyberattack.
"With the proper BC/DR plan in place, [geography] should be a non-issue, as consideration should be paid to ensure geographic redundancy has been put in place," Young said.
Making the right tradeoffs
"Essentially, you have to think about preparing for the worst as a risk management challenge," Bertrand said, adding that there is always a real relationship between the cost and risk associated with data loss and the cost of protection. It is especially important to look at and think about your most critical applications.
"If they are unavailable, it will cost you more and it will cost you more faster because they have so little tolerance for downtime," he said.
Essentially, you have to think about preparing for the worst as a risk management challenge.
Christophe BertrandSenior analyst, Enterprise Strategy Group
Risk calculation for different disaster recovery scenarios can range from an assumption of total security to an absurd example of preparing for the end of the world, Bertrand said.
"That event would have infinite consequences, but if the world actually ended, nothing you do would matter, so you wouldn't care," he added.
In the same vein, Bertrand mentioned the plausible but perhaps remote possibility of a nation successfully disrupting the entire global internet. Because essentially no one would be able to engage in business at all, should it matter enough to plan for it?
More realistically, there is, for every organization, a maximum threshold whereby the losses take the business out completely, he said. That could be a situation where, for example, you have only one data center that is in the middle of Florida and it is located below sea level.
"You would have an exposure much greater compared with someone that has two or three data centers in different locations with active replication," Bertrand said.
The point, he said, is that there are numerous risks and each has consequences. The end of the world is high cost but, given its low likelihood, it is also low risk. On the other hand, data corruption, loss and theft happen all the time.
Bertrand advised embracing a risk management perspective, then focusing on business criticality. Namely, what do you really need and when? That should be supplemented with some geography-related disaster recovery scenarios that tip the risk scales in one way or another. Regional blizzards, earthquakes or even a combination of such phenomena might be worth considering.
When you are regionally based or heavily exposed in one region, "it is important to have an out-of-region strategy," Bertrand said. Without that, even a comparatively minor event could become a big problem.
