Security incident response procedures: When to do a system shutdown

System shutdown scenarios

Shutting down a system in response to an information security incident is arguably the most drastic option that can be taken, but it might be the best option in certain scenarios. To determine whether a system shutdown is appropriate, organizations must look at the consequences of dealing with an incident versus shutting down the system.

Obvious scenarios that could justify shutting a system (or parts of a system) down could involve a threat to someone's life, or if the consequences from the incident could put the organization, or one of its customers, out of business. For example, if there were a possibility that an attacker could gain control of a computer system that regulates traffic lights, it would likely be best to disable that system, and thus the traffic lights, because drivers would hopefully know to treat the non-working traffic signals like stop signs. The consequences of allowing the attacker to control the traffic lights, on the other hand, could be dire.

These are extreme situations where the decision is easier to make, though; the more likely scenarios that enterprises could face will not be as drastic.

For instance, a system could be infected with a worm that is attacking other local systems, and removing the system from the network or turning the system off would prevent the worm from spreading to additional systems. Worms can spread quite quickly from one system to the next though, so deciding to turn a system off must be made rapidly. Such a decision would also depend on the security and availability controls implemented, including if there were controls in place that limited the compromise from spreading to the complete system versus just a compromised account.

In a system that contains no sensitive data and only has availability requirements, the security team could just do a basic calculation of the cost incurred from the downtime versus the recovery cost from containing and remediating the compromised system, and then make a decision based on the numbers. There are also scenarios where a full system shutdown might not be necessary. For example, shutting down an external network connection to prevent an attacker from gaining additional access while a high-value system is investigated is often a preferred, less drastic approach.

Of course, there are also scenarios where shutting a system down might be the worst option for responding to an information security incident. If an attacker has already compromised a local system, a system shutdown might destroy valuable evidence that could be used for forensics. Shutting down network connections, or possibly an entire network, could also destroy evidence that could be used in an investigation. It might be better to just unplug the network connection and investigate the system while it's still running, with the attacker unable to access the system. Individual enterprises would need to assess whether the potential value gained from such evidence would outweigh the need to shut down a system to prevent damage.

How to prepare for a system shutdown

Though shutting down a system is a drastic measure, there are some steps enterprises can take to prepare for such a scenario. First, gain a better understanding of what data is stored on the system and perform a business impact analysis (BIA). The BIA will document how important a system is to the business, what the system is used for and the impact of a potential outage. This will lead into developing a business continuity and disaster recovery plan (BCDRP), which is similar to an incident response plan in that they both need to be developed prior to an incident and periodically tested so if a shutdown becomes necessary, a playbook for dealing with the situation is on hand.

Before shutting a system down as part of a response procedure to a security incident, it's critical to get authorization from the appropriate parties. In developing the BCDRP or incident response plan, establish a channel of communication with the necessary people, including the chief information security officer, chief information officer, helpdesk, business owners and marketing so they will be able to quickly make an informed decision about potentially shutting down a system. If shutting the system down will essentially stop the organization's operations, senior management should be briefed. They will need to know the impact on the organization, the efforts already underway, potential costs and the impact of remediation. Who needs to know which details will vary depending on the organization and the available resources.

Finally, remember a system shutdown does not actually secure the system and should not be the last step in your security incident response procedure. After shutting down a system the next step is remediating whatever security issue caused the situation in the first place. Remediation efforts could include patching a system, making a configuration change or restricting access to only trusted connections. How long this step will take depends on the results of the BIA and BCDRP; systems with high impact from downtime should be remediated quickly though. For example, a Web server with a Web application susceptible to an SQL injection vulnerability might be shut down while a patch is being developed, a Web application firewall is set up or the configuration is changed to remove access by the Web server to run commands on the system.

Conclusion

For an organization under attack, the most drastic course of action is sometimes the best (or only) one to take. Understanding the business impact from downtime for a particular system or network will help determine whether resources should be devoted to protecting a system under distress or if a system shutdown is more appropriate. Regardless of which option is the best in a given scenario, ensuring a plan and communication channels are in place prior to an incident is critical to minimizing the impact on your organization.

About the author:
Nick Lewis, CISSP, is the information security officer at Saint Louis University. Nick received his Master of Science in information assurance from Norwich University in 2005, and in telecommunications from Michigan State University in 2002. Prior to joining Saint Louis University in 2011, Nick worked at the University of Michigan and at Boston Children's Hospital, the primary pediatric teaching hospital of Harvard Medical School, as well as for Internet2 and Michigan State University.