Follow Patch Tuesday best practices for optimal results

How to develop a patch management plan for Windows

A patch management plan is a policy that defines how and when your organization deploys newly released patches. For example, an organization might stipulate that all new patches must be deployed within 30 days, except for critical patches, which must be deployed within 15 days.

The IT department should use this buffer period between patch release and deployment to verify the patch's integrity. However, it is also important for those responsible for patch management to see what other organizations say about each patch. After all, there will always be sys admins who immediately deploy patches upon release. You can use the experiences of those early adopters to gauge whether a patch will likely cause problems. However, every network is different. Your organization can experience problems with a patch when nobody else has had issues. That's why it's so important to do patch testing.

The IT department should also make and maintain documentation for patch deployments, including a schedule for testing, approvals and rollbacks if a patch causes major problems.

How to build a systems and software inventory

Developing a patch management plan requires a comprehensive inventory of your IT infrastructure, including all the Windows Server systems, client machines and other Microsoft software, such as Microsoft Office, Exchange Server and SQL Server.

This is a difficult task to do manually, so it's a good idea to use software tools to help with the inventory creation process. Some automated tools that can assist include Microsoft Configuration Manager -- formerly System Center Configuration Manager -- PDQ Inventory and Lansweeper.

An IT infrastructure inventory should ideally include all IT assets. However, two inventory items are easy to overlook yet important from a patch management standpoint.

First, you need to know the software versions used throughout your organization. After all, it's impossible to know what patch testing you must do if you don't even know what software versions you are patching.

Second, you need to document any software dependencies that may exist. Otherwise, you may find yourself in a situation where patching an application breaks another seemingly unrelated piece of software. The best way to avoid such a situation is to include dependency applications in the patch testing process.

How to test Windows patches

While a comprehensive discussion of patch testing is well beyond the scope of this article, there are some Patch Tuesday best practices to keep in mind.

First, it's a good idea to create a virtual environment that mimics your production environment but on a smaller scale. That way, you can test patches on a sampling of systems that reflect your real-world infrastructure.

Configure the VMs the same way as their counterparts in your production environment. One option some sys admins use is to restore a production backup to a lab environment to duplicate the Windows systems.

Another best practice is to avoid the temptation to test patches just in your lab environment before deploying the updates to all other systems. No matter how hard you try, there will always be subtle differences between the testing and live environments. The best method is to test new patches in your lab. Once you are satisfied that the patches have not caused problems, deploy them to a representative sampling, and test the patches on those systems. If everything goes well, then you should be able to deploy the patches to the rest of the organization safely.

How to develop a patching priority list

Every organization has limited IT resources, and prioritizing patch testing and deployment is often necessary based on what a particular patch is designed to do. Some patches, for example, address obscure security vulnerabilities for which no real-world exploit exists. Other patches fix a vulnerability that was already exploited as part of a zero-day attack. Simply put, some patches are more important than others.

As you work to prioritize patches, there are two main things to consider. First, how much risk does the associated vulnerability pose to the organization? Higher risks need to be dealt with first. Second, are there systems that need to be patched first? You should patch a web server exposed to the internet before an internal server containing the same vulnerability because the external-facing web server is more likely to be compromised.

You may encounter situations in which patching is critically important, and there is pressure to skip the testing process. It's possible to justify this in some instances. Still, evaluate whether there are other actions that you can take to temporarily mitigate a vulnerability until you can properly test and deploy a patch. If your organization prefers to deploy patches as quickly as possible, it's imperative to have validated backups and an established rollback procedure to restore critical systems.

How to automate the patch deployment process

Manually applying patches is tedious, time-consuming and often prone to human error. As such, it's important to automate the patch management process. Automation is the only realistic option for applying patches to all the systems that need them.

Microsoft offers several tools to automate the patch deployment process, including Microsoft Configuration Manager, Microsoft Intune, Windows Server Update Services and Azure Arc. Several good third-party patch management tools are also available from vendors such as Ivanti, ManageEngine, NinjaOne and SolarWinds.

How to check for potential issues

Most times, patches do not cause any problems. It's important to check that patches are installed successfully. Verify patch deployments across the organization using reporting tools such as Microsoft Intune and Microsoft Configuration Manager.

Watching for signs of trouble after a patch has been deployed is important. An increase in help desk tickets is one indication, as well as notices from automated monitoring software.

Watch for abnormally high system resource consumption, particularly CPU and memory. You should also be on the lookout for unusual events in Windows event logs. Sudden sluggish performance can also indicate there was a problem with a patch.

The Microsoft Windows release health site is another resource that publishes known and resolved issues for Windows Server and desktop versions. It's also where Microsoft announces upcoming updates that might require sys admins to prepare their environment.

Follow these Patch Tuesday best practices to minimize disruptions

Patching is a difficult but necessary part of a Windows admin's tasks. There's no escape from Patch Tuesday -- with one rare exception in February 2017 when security updates were delayed when Microsoft paused the release of security updates to correct an issue with a patch.

To stay informed, regularly check Microsoft resources related to patches, such as the Exchange Server blog if you have on-premises Exchange, the Microsoft Security Response Center blog and the Windows release health site. The sys admin subreddit posts a regular Patch Tuesday Megathread on the second Tuesday of every month, where IT workers can share information about the newest batch of patches.

By staying informed, implementing a thorough testing procedure and establishing a reliable rollback method, sys admins can keep their Windows systems updated, while minimizing disruptions to avoid unnecessary stress.

Brien Posey is a former 22-time Microsoft MVP and a commercial astronaut candidate. In his more than 30 years in IT, he has served as a lead network engineer for the U.S. Department of Defense and a network administrator for some of the largest insurance companies in America.