Real-life change management examples prove it's harder than it looks
Whether your IT shop rigorously follows a service management framework or takes a laissez-faire approach to changes, these examples help hone change management procedures.
Change is scary; we get used to one way, and we know that changes normally don't go smoothly. In the IT world, both IT staff and end users get upset when things change. You'll never remove resistance entirely, but change management examples show that there are ways to lessen risks and get users on your side.
One example of change management occurs around configurations. Configurations change on a reasonably frequent basis, yet these updates can potentially do significant damage without proper due diligence at each stage of the rollout. A quick change that's applied with little thought and planning may work fine or it may not -- is it worth the risk?
IT Infrastructure Library (ITIL) and DevOps both have ideas around the lifecycle of products. Configuration changes are a slow process in a true ITIL philosophy, while DevOps relies on rapid and agile changes. ITIL and DevOps philosophies align on workflows though, with DevOps following a path of Plan > Develop > Test > Release, and ITIL similarly moving through Design > Transition > Operation.
The approach to implementing changes should revolve around these main points of a proper change management procedure. Configuration and other change types should be assessed on a case-by-case basis.
Understand the change
What are you changing, and why are you changing it? A team member must understand the reason for the change, and what the new configuration will do. Especially on large teams, it is unlikely that everyone knows the big picture. At least one team member, whether it's a technician or the project manager who initiated the change, must describe and advocate for it. Ideally, everyone involved knows why they're implementing the change, and should be able to agree that the change requested will have the expected outcome.
IT risk assessment surveys, such as this example from Pink Elephant, should be easy to complete and tell the team how big a change will be.
Reverse the change
Once the configuration change is understood along with its consequences, the IT team must consider a back-out plan to the change. In one change management example, switching an enable/disable setting in a Windows 10 group policy, backing out is as simple as toggling to the opposite option.
In any situation, the team must understand how that undo will be performed, and all the actions to take to roll it back. In this example, the Windows administrator may have a script ready to force clients to recheck their group policy settings once that toggle happens, rather than waiting for a 90-minute random window or a client computer reboot.
Test the change
A team member must understand the reason for the change, and what the new configuration will do.
The best way to verify changes is in a test/assurance environment identical to the production environment. Not everyone has that luxury, and it's difficult and time-consuming to maintain a 100% mirror configuration between test and production. For example, IT cannot duplicate server names on one Active Directory domain on the same network. Even without a fully separate, identical setup on identical infrastructure for the purposes of change management, IT teams still have plenty of options to test changes and reduce the risk of something going wrong.
Consider a change management example where a configuration must occur at the client end. Scaled testing is easy, because the IT team can target the change onto a single PC, and see if it has the desired outcome. If that is successful, they will continue to scale the configuration change rollout until everyone is confident the change has been fully tested.
Depending on how risky the change is, you can leave the change for a few hours, days or weeks before scaling it out. A canary segment helps to show problems, which may emerge from the change, that are not directly related to the change plan. For example, a Windows update required for Application A causes a new bug in Application B. With a change management plan based on limited initial deployment, the conflict cannot create widespread issues.
Implement the change
Returning to group policy as the change management example, consider the plan for a company-wide update on password complexity rules. From the user's perspective, IT should communicate and provide assistance to ease the change as it occurs. At the same time, the effect on a user population is gradual, as the new rules kick in only when a password expires. Change management demands good, clear communication to as broad of an audience as possible. This example change could be fine on the client side, but wreak havoc on the server side due to dependencies. For example, service accounts on a password rotation change talk to an old third-party application that only supports eight-character max passwords, and the new policy forces a minimum of 10 characters. The more broadly IT communicates changes, the more likely these issues under the covers will be foreseen and accommodated without creating outages or issues.
Monitor the change
Once the IT department rolls out a change, they need to find any issues. Without a feedback loop, how will IT make improvements for future changes?
From an infrastructure point of view, start by tracking normal usage before the change is applied to the production environment. Then, check operations management tools for issues such as high memory use from memory leaks, spiking CPU load or network throughput, and disk I/O utilization changes after the deployment.
Again, communication is important along with documentation covering what the change was and when it was done, for accountability and tracking purposes.
Too many changes
Doing things properly can take time, so how do you deal with having a backlog of changes, or having more changes than what you can handle? This comes back to the first step of the ITIL and DevOps cycles: planning and design. If needed, temporarily apply change freezes, allowing the team to go back to the proverbial drawing board.
Lots of small changes make sense in some scenarios, but not others. Spend time understanding the whole picture from beginning to end to make the best-fit choice. Change management examples show that IT shops might need a list of configuration changes, but they're deploying these configuration changes in too much of a piecemeal fashion, wasting time. In other scenarios, small individual changes are the best approach.
