pipeline as code

What is pipeline as code?

Pipeline as code is an approach to a continuous integration (CI) pipeline where the pipeline is expressed entirely in computer code. The entire pipeline, stored in version control, is expressed as a single script or program that can run with a single command-line execution. This may be best understood in contrast to traditional CI, which is configured through a user interface (UI), along with other as-code concepts.

What does 'as code' mean?

Computers automate straightforward, repeatable business processes. From processing insurance claims to monthly banking statements, computer programs take things in one state and create a new state. Virtual servers, Docker and cloud technology made it possible to create entire environments from the command line. Wrapping those in code provides logging, unit testing, and the possibility to handle exceptions and errors. That gave rise to infrastructure as code. Software-defined networking brought the same concept to the configuration of routers and the internal company network.

It makes sense to have the computer do the hard, repeatable work if the subject is a repeatable business process, but it is complex enough to have possibilities such as failure that require inspection, logging and notification. Instead of working with a UI to do the same steps over and over again, one could write a computer program, with libraries and configuration files, to do the subject-matter expertise, such as building infrastructure, defining network rules, running tests or running CI.

How is pipeline as code different from traditional CI?

Under traditional CI, deployment pipelines are configured through the UI. First, the build designer provides the software a source repository to check out the latest code from. Then, the designer clicks to add steps, giving each a label, such as build or test. This also tells the CI system how to recognize failure, who to notify and if that failure should stop the system from continuing.

With pipeline as code, the entire build pipeline is expressed in one configuration file or even computer program. Instead of setting things up through the front end, cutting and pasting -- or hand typing -- code to run at each stage, all of that is expressed in code.

Benefits of pipeline as code

Pipeline as code combines the ability of CI for logging, reporting, dashboards, trends, visualizations and notifications with the benefits of "as code," version control and programming. This aligns with the concept of DevOps by bringing modern programming rigor into the operations process.

Having the code under version control makes rollback and auditing simple. It's clear as to exactly who made what change and when they made that change. If a change to the pipeline causes problems, the person who made that change can get an alert, along with a link to that specific change. Pipelines developed as code have all the benefits of traditional source code, including looping structures, variables and access to code libraries, along with the ability to use object-oriented constructs, such as inheritance and encapsulation.

Drawbacks of pipeline as code

With pipeline as code as a computer program, it can become complex and difficult to debug. In general, the common practice is to try to keep pipeline code as simple as possible.

A second drawback with pipelines is that they lack the visualization component. Once defined, one cannot drag, drop or move them around. Modern UIs and plugs do exist to define the pipeline in a visualization and then export that as code to execute. Running the code and interpreting it in that visualization solves the problems of creation, debugging and transparency, all while still having a real pipeline as code.