Everyone wants a smooth patching process, but things can fall apart despite everyone's best efforts. Develop a solid recovery plan and test it before trouble strikes.
There are several general practices that can help improve the chances of rolling out Microsoft Exchange updates successfully.
Administrators apply updates to correct issues, but there are times a patch itself will break the system. There is no single way to update and -- if things do not go as planned -- recover Exchange. Every organization has different versions and configurations for the messaging platform. Every Microsoft patch is different.
This tutorial works off my earlier Exchange patching tutorial that covers several recommendations for a more predictable result. Organizations with a process that varies greatly from the guidance outlined in my previous article might want to consider implementing that method.
Make sure everyone knows the game plan
Even the best planned patching process can go astray. Exchange admins should emphasize this risk with management. When everyone knows problems can occur, but there is a set procedure to deal with potential issues, then our job becomes much easier.
A recovery effort is likely to get more chaotic when people are anxious. There is less chance of panic-induced mistakes when the organization knows about a contingency plan that covers most outcomes.
Read the release notes for Microsoft Exchange updates
Generally, there is a significant amount of information about both Windows and Exchange patches by the time Microsoft releases them to the public. Check through all the available documentation.
Exchange MVPs often have early access to these patches, and many of us put in a lot of effort to apply them in many different test and production settings. Take advantage of their experience and take some time to look through blogs that could save you a significant amount of time recovering your Exchange servers after a bad patch. Michel de Rooij's EighTwOne blog is a good resource for articles about Exchange updates.
Go slow and test along the way
Don't rush the patch process. Schedule Microsoft Exchange updates so there is enough time to deal with a recovery effort if something goes wrong. I generally start patching shortly after the close of business on a Friday night. This will give the IT staff the entire weekend, which should give enough time to test Exchange and ensure everything comes back online properly.
Schedule Microsoft Exchange updates so there is enough time to deal with a recovery effort if something goes wrong.
A well-planned patching process should mean that no resources are ever totally offline, but if redundancy is lost, then it's helpful to have enough of a cushion to recover the system to avoid disrupting users during a work day.
But there are only so many hours in a day. It might not be feasible to do a thorough test of the entire Exchange system after each reboot following the installation of a series of Windows and Exchange patches. But, at a minimum, take the time to check that all the necessary services will start on the servers after applying cumulative Microsoft Exchange updates.
Your testing methodology will evolve with experience. There are specific things you will learn are worth checking after each reboot, while there are others that can wait until the admins apply every patch and are ready to put the servers back into production.
Be sure to preserve redundancy
Properly deployed, Exchange is amazingly resilient. If the Exchange system configuration utilizes the platform's high availability features, then be aware which servers provide the redundancy for each other and patch those servers in an order that preserves that redundancy if a problem arises.
Prep Exchange to recover mailbox databases.
The Exchange Preferred Architecture encourages admins to set up all Exchange servers with the same roles deployed and the same services configured. In an organization with this arrangement, the patching order of the Exchange servers should not matter. If there are servers dedicated to client access roles and some dedicated to mailbox roles, then the patching order is likely much more significant.
Is it time to perform a restore?
Sometimes admins can roll back a patch or uninstall it to save a server from a rebuild, but that's not always possible. It takes time to build enough experience in Exchange administration to know when the only way to correct a problematic update is to restore the system from a backup.
Therefore, it's essential for any Exchange administrator to practice a restore procedure until it's no longer a foreign concept. Some IT tasks require time and preparation to understand fully, and a system restore is one of them. A trial by fire at 2 a.m. on a Saturday after a patch breaks the system is not the time to learn how to recover an Exchange server.
Lastly, know when to call for help, either through a support case with Microsoft or with a consultant. Everyone has a limit to their technical ability, so avoid unnecessary trouble by calling in some reinforcements when there is no clear way to unravel the issues.
Dig Deeper on Microsoft messaging and collaboration
