Deploy patches safely and sanely in a CI/CD workflow

Steps to deploy patches

One problem with CI/CD is that people can't agree on the precise stages the process should entail. Indeed, the CD part has dual meanings, referring to either delivery or deployment, depending on the implementation.

You can get an idea of the diversity of DevOps and other automation software by looking at the technology stacks shared by various companies on StackShare, a crowdsourced portal for sharing information about development tools. Regardless of implementation, the CI/CD process can be reduced to a few key stages:

1. code development
2. application testing
3. integration testing
4. acceptance testing and quality assurance (QA)
5. production

For patch management, the choice between delivery and deployment is critical. Delivery pauses the automation processes after QA acceptance, requiring a manual push to production. Deployment automates the entire chain, moving code to production once it's successfully passed the acceptance criteria for each stage of the pipeline. As we'll see, few organizations leave the entire CI/CD chain on autopilot since there's invariably a need for code reviews and a code-improvement feedback loop.

Finding a fit

The difference in how each approach handles the move to production illustrates a typical challenge with patch management in a DevOps environment: a disconnect between code development (Dev) and system deployment (Ops).

The DevOps methodology is supposed to bridge the historical and organizational gaps between the two. However, this culture change can't erase each group's history, which manifests in integration and delivery and deployment processes and tools that have evolved independently. Thus, an important step to achieving smooth DevOps patch management is joining two sets of tools and processes that weren't necessarily designed to work in concert.

This stitching isn't just technological. It's relatively easy to customize tools or to find new ones that can automate virtually any DevOps task. The harder and more important work is organizational: getting the Dev and Ops teams to collaborate and trust one another.

No matter how automated the DevOps toolchain, patches still require code review, lest they introduce a new set of bugs. This important review process can occur during unit and integration testing. For example, HotelTonight, an online travel booking site based in San Francisco, uses Solano Labs' Solano CI to automate its test and integration process. When a developer checks in code, Solano CI runs a test suite while other developers also review the code. Once the new code clears the tests and reviews, it's released to a staging cluster where -- depending on the feature -- it may face additional manual testing by QA before it is released to production.

Patch development philosophy

Aside from the process steps and degree of automation from code development to production package, another challenge in managing patches in a DevOps environment is how software updates are packaged and delivered.

Traditionally, namely in the era before VMs and cloud services, patches were code or configuration updates that only changed small pieces of a larger executable or library file. This approach was necessary because of the money, time and effort needed to redistribute an entire application. The incremental update approach remains popular, particularly for client software -- PCs, mobile devices -- for all the original reasons. However, it creates complexity for organizations forced to manage patches for multiple versions of an application on multiple systems.

Google provides an interesting wrinkle on the patch development and submittal process. This process is unique, owing to Google's massive size, but instructional because it illustrates some common practices. Google uses trunk-based development with a shared code repository to speed integration of new code. Automation scripts rebuild all code dependencies on every change. Should a change or patch cause major problems during testing, it's automatically backed out. However, given the size of Google's development team and the inefficiency of backing out changes, it has a presubmit tool that automatically analyzes changes and runs a set of basic tests before they are added to the trunk.

Google's code review process relies on segmenting the trunk into areas of code ownership, where teams review all changes to their area and ensure the integrity of the codebase. For large-scale patches, Google uses a custom tool called Rosie to split the change into smaller packages following the aforementioned code-ownership hierarchy. Rosie independently tests each, sends changes out for review and automatically commits them once they pass all tests and reviews.

An alternative philosophy to deploy patches incorporates the concept of immutable infrastructure. Immutable infrastructure is an application deployment strategy that capitalizes on the nexus of software development automation tools and software-defined infrastructure that can be programmatically configured and deployed.

In contrast to sending out binary patches or new VM images, immutable infrastructure uses the continuous deployment form of CI/CD to trigger an entirely new application deployment whenever new code is checked in and passes unit and integration tests and code reviews. In essence, immutable infrastructure moves patch management up the toolchain toward the developers and away from operations.

Netflix uses a variant of this technique with immutable servers, based on Amazon Web Services' Amazon Machine Images (AMIs). Once developers integrate new code using Jenkins, the CI automation software creates a new AMI in a test environment. Details of Netflix's testing regimen varies by application, but once an AMI passes integration tests, it's sent to a deployment tool (Spinnaker) for global distribution.

As was previously pointed out, immutable infrastructure solves a vexing patch management problem: tracking the myriad patch files needed to patch an application that has morphed into many different versions. Likewise, it eliminates the need to worry about system dependencies (i.e., OS and library version, system configurations, etc.) since these are all built into the source tree by treating infrastructure as code. This last requirement is a key impediment to using immutable infrastructure, namely the need to completely automate system configuration and deployment using templates, recipes and infrastructure automation software like HashiCorp's Packer and Terraform for image creation and Ansible, Chef or Salt for deployment.

Getting flow in the CI/CD workflow

Using CI/CD tools and processes allows organizations to more rapidly and consistently deploy patches, but only once they've spent the time to understand and define processes and a workflow that suit their needs. Aside from the guidance and examples already cited, here are some tips for building a smooth DevOps patch-management system:

• When building your CI/CD toolchain, explicitly define code review feedback loops and any steps requiring manual sign-off before proceeding.
• Incorporate automated deployment -- infrastructure as code -- tools into the back end of your toolchain, and wean operations teams from manual processes toward treating infrastructure as programmable resources.
• Determine if the immutable infrastructure model works for you. It's particularly appropriate for public cloud deployments.
• Cloud users should evaluate Google Cloud Source Repositories or available CI/CD services, such as AWS OpsWorks (Chef-based infrastructure automation), AWS CodePipeline (CI/CD service) and AWS CodeDeploy (Git-like code repository).

Using DevOps methodologies to integrate patch development and deployment processes into a CI/CD toolchain can significantly reduce the time to fix and distribute software bugs and security flaws, while reducing much of the administrative overhead of patch management.