Guido Vrola - Fotolia
Rethinking change control in software engineering
The traditional method of application change control won't fly in a modern, DevOps environment. Here's how we can rethink the process in a time where rapid app development rules.
Change control in software engineering is a formal fault-prevention process that requires development teams to request approval for any changes they make to an application or system. In a classic IT organization -- one comprised of an isolated operations group and an isolated development group -- change control can create unforeseen tension between these two teams.
When organizations mandate that their ops teams focus solely on stability, change control can quickly become change prevention, much to the chagrin of development teams that are mandated to continuously update and deliver new features. With DevOps now inverting the traditional IT delivery model, the question becomes: Can change control still work in the way it was intended?
It's likely that small, software-focused organizations running in the cloud won't use the term change control. They may just execute deployments when it makes sense, especially if the team doesn't yet charge for their services, or they have a way to turn a new service on for only a limited number of users.
On the other end, large organizations that still run COBOL tend to use monolithic ticketing systems to manage permissions and change approvals. However, most teams probably find themselves somewhere in the middle of these two extremes, leaving them in a place where they need to find a realistic balance between both the resiliency and flexibility of feature deployments.
Let's examine how to tweak change control policies in a DevOps world where stability and flexibility need to work in tandem and review practices you can implement to make this happen.
Decide what you can skip
It's possible for an organization to determine that some lower-level risks don't warrant evaluation through a formal system. By design, most DevOps processes already prioritize risk reduction and mitigation through automated rollbacks and feature flags.
A modern deploy system should leave a record of what changed, making the debug-and-fix process easy enough that change control isn't necessary. When you think about it, it's likely some companies already do this for mission-critical spreadsheets, blog posts and ad hoc queries, which often forgo formal version control.
Connect responsibility to authority
One of the negative aspects of change control in software engineering is that operations people fear they'll be blamed if things go wrong. In many cases, they will actually be on the hook to fix any change gone wrong. This problem rings especially true in an organizational structure where multiple teams contribute to a single consolidated deploy.
In contrast, imagine a world in which the team that makes the deploy is the one that lives with the consequences. This responsibility can become a reality when teams support their own features, and those features are deployed separately through microservices. When the only thing that can go down is search, suddenly every other person in the middle that's working on product display or recently viewed is much less concerned about failures.
High risk still needs formal control
When teams move toward continuous delivery, they tend to have a set of "what about" questions, such as the following:
- What about our SQL schema migrations?
- What about major upgrades to the web server or the operating system?
- What about changes made to mobile apps that have to go through an app store?
- What about code that affects personal or financial information?
If any of these concerns or any similar ones are a big issue, teams may want to continue using a more formal change control process. Of course, the stringency of these change control processes will depend on how risky your organization views a particular concern.
Prioritize notifications, not permissions
If the group can significantly lower the amount of time it takes them to identify and recover from problems, then organizations might want to consider morphing change control from a process of receiving permission to a system that creates a window for operations teams to examine already implemented changes for a period of time -- from a couple of hours to a couple of days. If needed, the developers can simply roll back the problematic change.
Put everything behind feature flags
Feature flags, which enable easy on-and-off configuration changes, make it possible to conduct so-called dark launches that generally do not require change control. If you think that feels like a bit of a logistical sleight of hand, you're right.
Teams can also combine this approach with the notification scheme outlined above, but keep in mind that this can be risky. Programmers that make mistakes with their conditional feature flags could accidentally deploy a change to production when it is supposed to stay dark, which means they might not be able to roll it back -- not easily, at least. The key to using feature flags is to place them where they make sense and to diligently make smart decisions regarding the risk they create.
Understand who change control affects
A key issue in change control in software engineering is figuring out who change control affects and how it affects them. If nearly everyone is affected by a change -- a likely scenario for teams contributing to a single mobile app deployment -- there tends to be heavy regression testing, triage meetings, go/no go meetings and documentation. This bureaucratic process often adds cost and delays, and it can be difficult to see where exactly the process provides value.
One way to cut away barrier-inducing change control processes is to isolate the impact of changes. Make it so that only the people responsible for making a change deal with any repercussions from that change. Then, it's on that team to figure out their process for change, not another team that may not even know a new feature or app was coming.
For companies that aren't quite where they want to be with change control, try gradually waning out old control processes for certain types of changes. These small transitions are the small victories that will lead you to a better overall change control system.
