How to recover from a DDoS attack
Learn how to recover from a DDoS attack and get operations back online quickly, while minimizing impact on customers and brand reputation.
Your organization was hit by a DDoS attack. It has been able to detect and stop the attack. Now what?
It's time to recover. Key steps include restoring services and conducting a post-attack review.
Step 1. Restore services
Restarting or bringing applications and services back online after a DDoS attack requires a sound knowledge of how the apps and systems work and interconnect. Teams need a methodical, documented roadmap of the correct restoration sequence. Without one, teams could face cascading failures. For example, one system might not function properly if supporting services aren't back up yet, which, in turn, affects other services, making the problem worse.
As services come back online, they might experience a flood of genuine connection attempts by users trying to reconnect. Sometimes, this can create an application layer DDoS effect, forcing everything offline again. To counter this, one simple option is to lower rate and connection limits to levels teams know the system can cope with. A better option, if available, is to route traffic to different data centers based on IP address ranges or geography.
If the organization's ISP has dropped connectivity, it needs to get it restored. This might require explaining what protections have been put in place to handle another attack.
If it was a Layer 3 or 4 DDoS attack, then run the clear ip bgp * command on Border Gateway Protocol routers. This reestablishes BGP connections so user requests are correctly routed to services and services no longer appear offline. IP transit providers and peering partners have flushed routing information about 90 seconds after the attack started.
Step 2. Conduct a post-attack review
After services have been returned to regular operations, security teams need to work with stakeholders to complete a damage assessment and a lessons-learned exercise to better prepare for the next attack, which will surely happen eventually.
The direct costs of an attack include lost revenue and production, as well as hosting costs, such as additional cloud instances. These expenses depend on how and for how long services were affected. Costs might also include indirect or internal costs, such as lost productivity, customer complaints, negative press or reputational damage. Note that users might have also been affected by the attack because of defensive measures that prevented them from accessing services. It's important to put a value on these costs because they help determine appropriate DDoS mitigation budgets going forward.
Focus the lessons-learned exercise on evaluating how well existing defenses and response procedures worked so any weaknesses or shortcomings can be rectified. Log data from internal network devices and applications, as well as relevant third-party providers, should reveal which attack protocols and patterns were used, the length and peak amount of network data and requests recorded, and which assets were targeted. This information shows what additional mitigations are needed to handle a similar attack.
Gather the following key metrics to help understand how well or poorly defenses and plans worked:
- Time to detect measures how quickly the attack was detected.
- Time to alert measures how soon key stakeholders were notified of the attack.
- Time to divert measures the time it takes for security controls to initiate traffic blocking and diversion to avoid downtime.
- Consistency of mitigation measures the percentage of malicious traffic allowed through -- a key measurement of the effectiveness of defenses.
These metrics also help measure how well existing DDoS mitigation providers or other relevant third-party vendors fulfilled their service-level agreements.
Another key statistic to consider is the ratio of legitimate traffic to malicious traffic that was stopped. This information helps identify weak spots in defenses that allowed a particular type of attack, attack vector or attack pattern to succeed or left specific services more susceptible to being disrupted than others. Address any shortcomings as soon as possible by upgrading tools, modifying trigger points and increasing network resources.
Also, consider how well communications with stakeholders was managed. Was it timely and informative? Contradictory messaging can frustrate and alienate users. In the future, designate one person to be responsible for coordinating updates over all communication channels. The legal team might need to file a report about an attack with industry regulators, law enforcement agencies and its cybersecurity insurance company. Also, consider advising the FBI's Internet Crime Complaint Center to help it better understand current attack techniques to help others defeat future attacks.
Even if an organization doesn't experience a full-blown DDoS attack over the course of the year, it's important to refresh impact forecasts and DDoS protection plans regularly. Review current trends, products and services annually to ensure IT environments are DDoS-resilient and that the best DDoS attack detection tools are in place and correctly configured and tuned.
Michael Cobb, CISSP-ISSAP, is a renowned security author with more than 20 years of experience in the IT industry.
