How to achieve continuous feedback in DevOps pipelines
Continuous feedback is critical to optimize DevOps processes. While tools can enable feedback loops, DevOps teams still must reduce unnecessary noise and continually measure success.
Software development is not a one-way street, which is why the DevOps loop includes code creation, test, delivery, deployment and feedback to start the cycle anew. Organizations achieve the DevOps loop with CI/CD, and CI/CD thrives on a continuous cycle of feedback.
Continuous feedback doesn't just occur at the end of the software development process. It requires coordination between teams throughout code development, deployment and support, even in non-IT functions, such as marketing.
Editor's note:Learn more about some CI/CD tools that can support DevOps feedback loops, with this related article.
CI/CD methodologies and feedback
CI/CD phases typically build on each other. DevOps organizations can implement some or all aspects of CI/CD, and the best approach varies from team to team and project to project.
Continuous integration is the software development practice of constant assimilation of code changes to a common code repository. CI encourages smaller and more frequent code changes than other development methodologies, which leads to rapid code iteration and testing.
When practicing CI, the team focuses on whether a build and set of tests succeeded or failed before moving on to the next part of the delivery process. Generally, it works like this: A developer commits a change to a version control system, which communicates with the CI tool. At that time, the pipeline triggers tools to compile or package that code, which is then tested to verify whether the build succeeded. The development team takes action based on whether the build progressed correctly or failed.
Tested code is then delivered for packaging and deployment to production. Continuous delivery means that the executable code is built and ready to go as soon as CI completes. The QA team communicates with developers, or it receives an automated message indicating the status of the build. In this stage, the decision to deploy to production is normally a manual one.
For continuous deployment, DevOps teams implement a set of automated steps to take code live. Continuous deployment is one of the hardest practices to get proper feedback on. There are continuous feedback sources in production, generally server metrics, application performance metrics and customer feedback. Though code might successfully deploy into production, the automation and monitoring setup might not give actual feedback on the effects each release has on end users.
Editor's note: In the abbreviation CI/CD, CD typically refers to continuous delivery, not the more complex continuous deployment.
DevOps feedback in real life
The feedback generated by the various systems in a DevOps environment is difficult for many organizations to manage and decipher. Even worse, wrong or incomplete information can quickly waste teams' time on troubleshooting.
Integrate real-time communication. Many organizations use email to communicate between teams. But chat clients offer a quicker back-and-forth that suits fast-paced and automated DevOps pipelines. Most chat systems integrate into CI/CD tools, which makes it easy to send status messages and other feedback. But messaging can be lost in the sheer volume of chatter and alerts. Look at what is essential to convey about CI/CD processes, so team members just get what they need.
Focus messaging to cut through noise. To focus feedback and prevent unnecessary noise, invest the time to determine exactly what a DevOps team needs in a feedback message and how quickly it needs to know about a given result, such as success or failure or automated action. Reduce unnecessary, repetitive messages by rolling them up into summaries.
Don't totally abandon manual messages. Manual steps still have a place in the world of continuous automation. For example, a QA team finds a bug in a build and, because of an approval requirement in place, it must halt progress. It can provide feedback to the developers, indicating and explaining the reason for the failure. With this input, a developer can update the code, which then makes it back to the QA team in a fixed build. With this feedback loop complete, and if all QA tests pass, the build moves forward to release into production.
Emphasize system monitoring. Monitoring is a critical component of DevOps deployment, and teams can get valuable feedback to developers from monitored metrics. Monitoring acts as the success or failure indicator for updated code at the end of the CI/CD process, completing the DevOps feedback loop for the next iteration to begin. Look for monitoring tools that enable users to make a point-in-time notation regarding a build release.
Monitoring tools can play a role in continuous feedback in DevOps pipelines that extend into production. For example, a monitoring tool is set to generate an alert for variations in CPU and memory usage. When developers and QA testers get this feedback on resource consumption, they can act upon the information to address problems with the build or further optimize the code.
Expand beyond traditional feedback
Beyond these expected continuous feedback sources in DevOps deployments, look for other useful types, such as marketing, documentation and support. IT teams might not seek feedback in these places because they fall outside of the CI/CD process, but they can be just as important and extend the loop from start to finish.
Look for ways to incorporate alerts from these sources into the feedback process. For example, an uptick in support requests is useful information about a release. You need a way for support representatives to indicate there is a problem. In ITIL-based IT service management approaches, for example, the alert would come in the form of a problem record, which is linked back to the last build.
Another example shows the value of documentation for feedback loops. As code moves through the CI/CD pipeline to testing, actionable messages are sent to the documentation team where it can tag code changes as new or enhanced functionality, and get the documentation up-to-date the moment code goes live and requires support.
Customers need to know about new features and functionality, which makes marketing part of the DevOps cycle as well. The marketing team should be included in the DevOps process to prepare appropriate materials, timed around the production release. The more automated feedback and communication, the easier it is to keep these disparate groups working together.
Measure feedback success
Continuous feedback in DevOps pipelines is only as good as the systems put in place to achieve it. You risk adding a lot of complexity for little gain. Use DevOps metrics to tell if the feedback loops save time and increase productivity. Measures to watch include mean time to resolution (MTTR) and dev, fix and full cycle time.
MTTR is the time it takes for feedback to lead to a fix. Typically, feedback comes in the form of a customer support request, which gets channeled to developers to work on, but it could also be automated alerts from monitoring tools. A support team fields the bug reports about an issue with a new release, creating a problem record for that build. The problem report gives developers feedback on what went wrong, and they apply their expertise to change the problem code. The time from initial reports to when the DevOps team releases a new version to production comprises TTR. Teams can compare each problem fix to the MTTR to identify bottlenecks and other process problems and should aim to reduce overall MTTR as an ongoing goal.
Continuous feedback in DevOps pipelines is only as good as the systems put in place to achieve it.
DevOps is about moving fast, but that speed is for naught if it culminates in a failed build that takes a long time to fix. Developers receive the feedback that a build failed rather quickly, but that's just the start of the clock. DevOps teams should track how long it takes to fix the problem. The more information they have on the build failure, the better equipped they are to resolve it. Long failed-build fix times hold up other development processes and must be addressed, potentially with training to write better code.
Full sprint cycle time, also known as commit-to-deploy time, provides information on the health of a CI/CD pipeline. Many DevOps teams are founded upon Agile software development approaches, including the use of sprints.
A sprint puts strict time limits on a continuous integration and delivery cycle, forcing the team to identify slow steps or areas where code waits for a tool or person to take action on it. Feedback loops contain valuable information about slowdowns within the sprint cycle. Within the full sprint cycle time, also measure feature development time as a stand-alone metric. And mature DevOps organizations should look beyond IT into documentation and marketing efforts as included in the measure of cycle time.
Track these metrics and others to guide and gauge improvement in the CI/CD pipeline over time. If you're new to CI/CD, measure everything possible about development and deployment systems before and after implementing it. If you find that new processes don't save time, then take a step back and evaluate where processes break down.
