Google's DORA DevOps report warns against metrics misuse

Organizational pitfalls: the perils of metrics misuse

It's natural for engineering managers and executives to use DORA metric results as goals and to compare the performance of different development teams, but this is a mistake, said Nathen Harvey, developer advocate at DORA and Google Cloud.

Nathen Harvey

"What I really want leaders to do is not celebrate the fastest software delivery teams," he said. "I want them to celebrate the most improved software delivery teams at the end of the year. Let's look at who improved the most. Because that's the team that's embracing this idea of continuous improvement."

Continuous improvement is never "done," which can be difficult for business leaders to absorb, Harvey added. Even the most improved team can still improve further. The slowest team within a company might have improved the most based on the specifics of the application it delivers. Comparing its metrics to a team that develops a different application -- with different constraints, infrastructure requirements and user experiences -- often isn't productive and can even be toxic, he said.

The DORA DevOps report calls for software development teams to focus not on being "feature factories," in Harvey's words but on user experience and team wellbeing.

"Engineers might not know who they're building for [or] why they're building these features, they're just told, 'Ship more, ship more, ship more, ship more,'" he said. "What we see in those sorts of teams is higher burnout -- even though they're able to ship faster, they're maybe not shipping the right thing."

Teams that build software with the end user in mind, however, have 40% higher organizational performance, according to the DORA DevOps report. The ideal approach is a balance between deployment speed, operational performance and user experience, the report states.

One team's experience with DORA metrics automation

This year, Sleuth.io and Propelo, acquired by Harness in January, took further steps to make use of DORA metrics -- not just reporting on them but allowing them to trigger automated workflows to enforce best practices. Propelo's integration into the Harness DevOps platform means users can automatically trigger actions in CI/CD pipelines based on DORA metrics.

Sleuth followed suit with the addition of Sleuth Actions and Sleuth Automations last month. Sleuth Actions was the framework the vendor developed to automate IT processes. It has been expanded and renamed Sleuth Automations, a set of pre-packaged workflows for third-party systems, such as GitHub Actions, that are offered through the Sleuth Automations Marketplace.

Cobre, a corporate payment platform provider in Colombia, began using Sleuth to report on DORA metrics about a year ago. It uses Sleuth Automations to trigger Slack notifications if updates lag between QA and production as well as to automatically block pull requests (PRs) in GitHub Actions if they don't meet policy requirements.

"It doesn't allow a developer to merge a PR if it has more than 20 files changed. It's too big," said Juan Matheus, solutions architect at Cobre. This enforces the DORA-recommended best practice of making small, frequent changes rather than large updates to software.

"This can also help you to encourage your developers to push code to production faster, because they know they can't accumulate a lot of changes," he said.

As this year's DORA DevOps report suggests, applying DORA metrics can lead to a process of continuous learning and improvement. A common bottleneck identified by this year's report, slow code reviews, surfaced for Cobre as it monitored DORA metrics.

There's been a process of continuous learning behind collecting data to measure DevOps team performance, even with a tool such as Sleuth, said Manuel Sanabria, product delivery director at Cobre.

Fixating on performance metrics can lead to ineffective behaviors. Investing in capabilities and learning is a better way to enable success. Teams that learn the most improve the most.

Specifically, change failure rate and MTTR have been tricky for Cobre, in terms of knowing what data to collect and translating raw data from the company's New Relic observability tool into DORA metrics, he said.

Sleuth's co-founder acknowledged the difficulty Cobre faced.

"How each team defines failure is unique," said Dylan Etkin, co-founder and CEO of Sleuth. "When a team chooses to use custom metrics, like the Cobre team, it can take some configuration on the team's part to decide exactly what is a relevant metric and to understand if that metric truly represents failure for their teams or projects."

In fact, DORA agrees that MTTR has been a tricky statistic, which is why this year, that metric was reworked and renamed failed deployment recovery time, according to Harvey.

"If I push a change to production and that causes an incident, that's one type of failure," Harvey said. "If a backhoe comes and cuts the power to my data center, that's a different kind of failure. And what I need to do is very different in both of those scenarios. So we really had to focus exclusively on those failed deployments."

Still, because each DevOps team and organization is different, it's difficult to prescribe a specific procedure for collecting data on these metrics, Harvey said.

"Some teams will look at, 'Well, if there's a release …' and then within a very short time they have another release, or see a rollback, we can assume … that first release was a change failure," he said. "But it's not necessarily something [where] you can look at your version control system and get the data out of that."

Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.