Part of:Fold security teams into DevOps: Get DevSecOps
Patching best practices IT ops and security can agree on
Patches are necessary for maintenance, but can often cause a rift between IT ops and security teams. Follow these best practices to unify expectations and keep tension at bay.
Everyone knows OSes and applications require patching, but the debate surrounding frequency can be contentious. This cadence is also the practice on which IT ops teams and security often differ in opinion -- but it's not because one wants security and the other doesn't.
Differing opinions are often productive in IT teams because it presents a range of perspectives and opportunity. However, it can result in a finger-pointing scenario that hurts everyone. Who conducts the patching process and how often it's done is a workflow that everyone must determine and support.
For IT operations professionals, what does this close collaboration with the security team look like?
IT and security teams are colleagues, not enemies
Security teams ensure the business and its applications, data and infrastructure stay secure; this is a unilaterally agreed-upon necessity. The danger is its effect on the cost of doing actual business.
Security teams often prioritize critical protection, like that new simple 24-character alphanumeric complex password they want you to use. The same challenge holds with patching: Many security teams will insist that every patch is critical and should be installed as soon as possible, with no exceptions. But the IT ops team is responsible for the work too.
It's a situation that can easily lead to IT operations staff feeling unheard or resentful, as they are the front line for customers' and users' -- often angry -- feedback when systems are taken down or production is disrupted for patching efforts. There must be a balance between enforcing security measures and conducting traditional business tasks, because both tasks and teams are related -- and reliant on each other.
Establish a collaborative framework
Pursue this balancing act one step at a time. One of the key things to determine is how to handle the patching process. This drives much of the framework because it sets the time window necessary to complete patching and outage recovery. It's not as simple as it might sound. Be clear with this process and the responsibility hierarchy -- and document all decisions -- because this cannot be taken for granted.
For example, application owners might give you mere minutes for an outage window instead of a realistic time frame in which to work. Or the management team might require that each patch is installed separately and then tested before starting the next patch, which can slow Windows update efforts into days rather than hours. It might sound ridiculous to an IT professional with experience in the topic, but for many others, the update process for mobile devices, such as phones and tablets -- namely apps that update in seconds -- has created a warped view of how software works.
It's unlikely that an IT team needs to collect every single patch note, but create a general framework of both patching and any additional steps to perform after patching a server. For example, IT teams could include the steps to start services or processes to ensure the application is back online.
Set unified expectations
Once IT operations teams define the framework and processes, they must set expectations for both security teams and end users. The patching cadence is often the first consideration, but this is actually a secondary aspect.
What drives the cadence is the testing level required after installing a patch. If testing involves only a few key applications, the cadence can be rapid; if you need to test hundreds of applications, however, that cadence slows to months between patches. And what determines that aspect lies in the effect on business operation.
If the system is known to go offline from 1 a.m. to 2 a.m. on the seventh day of each month, people can plan around that; give them that opportunity to prepare.
Consequently, most organizations settle into testing key applications after a patch instead of an exhaustive run through all applications. There is a chance that a bug will get missed in this practice, but reducing testing time counters the risk, which increases the patch frequency -- which security supports. This balance between the groups is key: It's compromise for all parties involved.
Define exceptions
The exception is an instance of a critical issue, for which response must be security-driven and performed differently than normal updates. However, the software vendor determines the definition of critical, not the internal security team. The latter opens the door to abuse of the critical patch flag, which muddies the waters.
An all-hands critical patch installation should be a rare exception, never the norm. Normal update scheduling should be set in a monthly or quarterly meeting, and supported by reminder systems that become part of the process. Once this full process is established, it's easier for all parties involved to know when the outage is both coming and completed. If the system is known to go offline from 1 a.m. to 2 a.m. on the seventh day of each month, people can plan around that; give them that opportunity to prepare.
Unexpected outages generate frustration, but planned outages are a nonoptional -- and expected -- part of life for the end user. Patching becomes a simple speed bump that few pay attention to, which is the ideal scenario for IT operations teams. Outage windows and updates that go unnoticed -- or unacknowledged -- are indicative of a successful system. Altogether, this collaboration, cooperation and compromise among IT operations, IT security and business leaders should result in a patching cadence that satisfies all three groups.
