12 CI/CD best practices for DevOps teams

1. Set priorities for security, testing and time to release

A CI/CD pipeline might provide more secure software, more automated testing or quicker time to release. If you have a slow, manual process in place, you might be able to achieve all three, but more than likely, these improvements will conflict with each other. To get the most value out of your pipeline, prioritize the most important benefit, and focus on tools and processes that grant that benefit.

For example, if your main goal is to release software more quickly but you also want to improve security, consider creating separate automated processes in your pipeline for each. You can have a primary pipeline that builds, tests and releases your software as you develop new features. Alongside this pipeline, where you don't slow down the main development process, you can also run thorough automated security scans of your applications to spot vulnerabilities.

Ultimately, it's important to evaluate and consider the potential impact of changing priorities and workflows -- and ensure that any impacts still align with business goals -- before implementing changes.

2. Define specific goals for the CI/CD pipeline

Before you begin your project, ask this question: What do I hope to accomplish by using a CI/CD pipeline? The answer might be shorter time to release, better software quality or increased automated testing. These goals should all be part of a checklist for CI/CD adoption.

Clear goals also inform all members of the team and provide a reliable North Star to keep everyone focused on choosing tools and creating automation that accomplishes these goals. The team also can better estimate the work needed to achieve those goals.

Do not underestimate the importance of goals. Although all DevOps implementations follow similar principles and patterns, there are countless possible variations in tooling, development processes, deployment strategies and metrics. The same DevOps deployment may not work the same way for any two organizations.

3. Use the pipeline to foster collaboration

Despite the emphasis on automation in a CI/CD pipeline, it doesn't start out completely automated without some manual intervention. Changes may take weeks to months of work to add. Moreover, DevOps concepts, such as continuous integration, continuous delivery and continuous deployment, are a mindset -- staff members need time to adjust to the new processes from a technical standpoint but also need to see the benefits and build trust in automation.

Encourage development teams to share ownership of the CI/CD pipeline. It's easy to fall into siloed patterns, in which developers throw code changes over the fence to testers, who then toss the code to an operations team for deployment. Ironically, automation without collaboration defeats the purpose of DevOps. Without shared responsibility, bugs take longer to fix, and overall quality decreases. When tests fail in your CI/CD pipeline, have developers triage the failures and correct the tests or the code if needed. Testers should help guide developers on best practices, but it's more important to share the burden of maintaining the tests.

Although all DevOps implementations follow similar principles and patterns, there are countless possible variations in tooling, development processes, deployment strategies and metrics. The same DevOps deployment may not work the same way for any two organizations.

4. Choose tools that support your priorities

There are many tools available to help build your application -- automated UI testing, security testing and more. To start, automated build systems provide a remote, ephemeral environment where you can build and test your application code or run any other automated tool that works in a command-line system. Popular examples include Ansible, Bamboo, Travis CI, Jenkins, CircleCI, GitHub Actions and GitLab CI/CD. All of these use a configuration file, where you specify commands for your application's pipeline. They also integrate with Docker containers, which ensure your build environment is properly set up with the dependencies needed to build and test your application.

Popular testing tools include Selenium, Cucumber, Test Studio, Jasmine and TestComplete. Some mobile app testing tools, such as BrowserStack and Sauce Labs, provide remote machines specifically for running tests, which can speed up your test suites significantly. If compatibility testing is of high priority, tools such as these make it easy to run the same test across many different devices, screen sizes and platforms. Other widely available automated testing tools -- Cypress is a popular choice -- also provide a testing framework, as well as a remote environment, to run the tests.

Testing tools can be an economical way to fortify your pipeline without writing a lot of custom code. Remember that it's often best to use an existing solution rather than take the time and energy to create something from scratch. For example, using a tool with preexisting templates can be far more efficient than building scripts from the ground up.

5. Include the entire team in CI/CD implementation

This is an extension of collaboration. Rather than implementing the CI/CD pipeline with a small team, encourage all team members who interact with the CI/CD pipeline to contribute something. Many small tasks arise throughout the process, including feature requests once a team gets familiar with the pipeline implementation. Group involvement helps introduce the team to the CI/CD process from the start. Wider collaboration avoids a situation where development is slowed or stopped by something that only a handful of people can fix -- or even understand.

Developers, testers and DevOps engineers working together can use automation to replace or augment processes that ensure the pipeline provides value. Collaboratively design a pipeline from a high level, and then dole out the work evenly among teams. DevOps engineers should provide guidance on best practices, but at the end of the day, the CI/CD pipeline serves the development and QA team as its ultimate user. The more ownership the team has, the tighter the feedback loop is between bugs and fixes.

6. Build once, test repeatedly

One way to create a more efficient CI/CD pipeline is to create a project's build artifacts only once. For example, if your application deployment uses a Docker container, build the Docker container at the beginning of your pipeline, and reuse the same container for all subsequent deployments and testing. This doesn't just save time -- it also ensures continuity across deployments and avoids any issues with dependencies pulled in during the Docker build.

In addition, rebuilding the software for each new stage in the pipeline or each different environment can lead to inconsistencies in the codebase, which create serious problems with compatibility, testing, version control and deployment.

7. Mock services in development environments

One item that can hinder tests is rate limits imposed by third-party services. For example, an application might use a third-party authentication service, such as Amazon Cognito or Auth0, to register and authenticate users. In production, these services work well because they are expected to handle production levels of load. In development, however, these services' endpoints are usually rate-limited to avoid costs associated with handling production levels of load. Consider Stripe's free "test" mode -- the company still provides bandwidth and incurs costs to respond to requests from this service. Without rate limits, it would be uneconomical for the company.

You could pay for the same level of service in your development and production environments, but you pay at least twice as much to cover them both. A more economical strategy is to mock these third-party services and provide a feature flag to enable a fake implementation of the service. You don't face rate limiting, the service always responds successfully, and your load and performance tests can more accurately measure your code's performance. Don't forget to test with the real integration before shipping to production.

8. Use ephemeral, scripted environments

It is possible to have a CI/CD pipeline with static environments where builds are installed as a part of CI/CD, but dynamic environments make more sense. Having one environment in CI/CD means you can deploy and test only one build at a time. Dynamic environments enable your pipeline to scale to as many builds as you like. Tools such as Docker Compose or Kubernetes make dynamic environments faster to configure and launch compared to cloud compute servers alone.

In addition, dynamic environments can provide more flexibility and versatility to accommodate a wider array of project types and requirements. Use scripts to define the needed workflow or process. For example, different types of projects might benefit from different types of testing or test data sets, which can all be defined through scripting associated with the project.

9. Fail fast

Design your pipeline in such a way that you can quickly find major defects. Collect a number of tests that validate your application's most important features, and run them on new deployments prior to more intensive testing efforts. This way, you invest minimal time to confirm a deployment is worth testing more thoroughly. Defects found soon after development are often easier to remediate than those discovered after developers have moved on to other tasks.

Key factors here include observability and metrics, enabling developers to see inside of the process and receive prompt feedback that facilitates rapid remediation. Observability and metrics are tightly related to tooling and process workflows and should always be discussed together.

10. Think small

A central weakness of traditional Waterfall-style development is its all-or-nothing approach. DevOps and other Agile-style development paradigms are attractive because they focus on small, rapid, incremental changes. Keep each change small, and commit changes frequently. Each commit triggers new builds and relatively quick tests, including unit, blue-green and canary tests. Successful tests can then be merged quickly back into the main branch. This enhances overall development, while limiting version and branch control.

11. Refine the deployment strategy

An important part of the workflow is in the end delivery and deployment stage, where a tested build can be available for deployment. However, software delivery does not obligate deployment -- or deployment as a regular part of the process. DevOps teams must choose between continuous delivery -- in which a completed build is delivered for deployment -- and continuous deployment -- in which validated builds are automatically deployed to production.

DevOps teams must build a strong knowledge of deployment strategies. This includes the CD tools, resources and policies, such as cloud utilization guidelines, that should be observed during a deployment. In addition, DevOps teams should create a detailed plan for instrumenting deployment to enable observability and gather performance metrics, as well as provide a clear rollback and recovery plan if issues arise once a new build is deployed.

12. Take advantage of automation

Complex and lengthy workflows, combined with frequent iterations, make DevOps pipelines extremely difficult to operate manually. The chances for delays and errors caused by human intervention are too great. Consequently, DevOps benefits from high levels of automation. Automation starts with building, artifact management and version control; extends through testing and validation; and continues through to staging, deployment, and metrics gathering and reporting. A mature DevOps process automates almost all the pipeline once a code commit is initiated.

Building a CI/CD pipeline is an ever-evolving process with many different paths. It is certainly worth the effort to build a pipeline -- even more so, a pipeline that is efficient and valuable. Following these tips helps you create a process that improves the quality of your projects and saves you time.

CI/CD mistakes to avoid

The promise of CI/CD can be compelling, but it can also lead to some unexpected pitfalls. Such challenges can limit the effectiveness of a CI/CD initiative, impair project outcomes and defeat the purpose of CD/CD initiatives in the first place. Common CD/CD mistakes include the following:

• Improper or incomplete automation. Effective CI/CD depends on comprehensive and well-planned automation workflows to shepherd each iteration from coding and building, through testing to staging and deployment. Partial or incomplete automation -- especially between pipeline tools that don't integrate well -- can lead to CI/CD process problems.
• Incomplete or omitted metrics. CI/CD pipelines offer opportunities to objectively measure the performance and outcomes of the pipeline, such as the number of code commits per day. Development managers and teams must understand the metrics that are important to the project and strive to implement suitable instrumentation and reporting. The goal is to validate CI/CD performance and look for opportunities to improve.
• Inadequate resources. Successful CI can involve multiple developers -- even multiple teams -- all working on the codebase simultaneously. It's possible that too many hands on the codebase can reduce development efficiency. For example, some code may be checked out, preventing others from working on it. In other cases, frequent CI jobs can demand significant resources. Efficient CI/CD can often depend on finding a suitable balance among the size of each job or commit, the frequency of each job or commit, and the resources needed to process each job.
• Conflation of delivery and deployment. Continuous deployment can intimidate many organizations, while continuous delivery can be a welcome outcome -- but both are sometimes termed CD. Teams should design the pipeline to support deployment, but there should be no obligation to continuous deployment unless the CI/CD pipeline and team are mature enough to support the added demands of deployment.
• Lack of collaboration. All Agile development paradigms tout collaboration as a key factor, but it's easy for automation to spawn unintended new silos. Instead of teams "throwing things over the fence," it's the automation workflow that enables the same result. It's important for development teams to maintain their collaborative efforts throughout the pipeline, no matter how mature the process becomes.

Matthew Grasberger is senior cloud engineer at Dynatrace. He has experience in test automation, DevOps engineering, security automation and open source mobile testing frameworks.

Stephen J. Bigelow, senior technology editor at TechTarget, has more than 20 years of technical writing experience in the PC and technology industry.