Sergey Nivens - Fotolia

Opinion

Are you using the right DevOps metrics in your organization?

It's tempting to think it's all about how fast you can deploy in DevOps, but the reality is, it's just one of the factors to consider. Mainline branch stability matters the most.

Valerie Silverthorne
By
Published: 23 Feb 2018

If you had to deploy right now, could you?

That intriguing question forms the basis for a recent survey by Circle Internet Services Inc., aka CircleCI, looking at DevOps metrics related to deployment and readiness. A CI/CD platform maker, CircleCI paired with Alexa Internet Inc.'s global internet ranking service to see if there was a correlation between DevOps success and business achievement.

At a time when most large companies around the world are at least considering this new methodology, these results offer a unique insight into the DevOps metrics -- and strategies -- that might matter most.

For most companies, DevOps is definitely a work in progress. "Even within our user base, there is a wide variance in the sophistication and maturing of people's [DevOps] practices," explained Jim Rose, CEO of CircleCI.

Mainline branch stability

When it comes to deployment, the key factor is mainline branch stability, because that tells a company how quickly and how often to continuously integrate, Rose said. According to the survey, 80% of organizations keep the mainline branch stable 90% of the time. Highly ranked Alexa companies, those in the 95th percentile, kept their mainline branch stable 99.9% of the time. "Lots of people get over their skis and want speed, speed, speed," he said. "We want you to move fast, but if you're not in a good stable place to do that, you can inject a lot of chaos into the project."

Jim RoseJim Rose

The report has some ideas to make the mainline branch more stable. Best practices include healthy test suites, the use of feature flags and a solid strategy for failure recovery. Another suggestion is to make sure the entire team understands the costs of failure. Rose noted that some companies actually send DevOps teams to the field to watch end users and develop a real-world understanding of the problem.

DevOps deployment frequency

DevOps metrics related to how fast a company deploys are also linked directly to business success. The survey said 80.2% of organizations deploy in under 15 minutes. But the top 5% of the Alexa ranked companies deploy in an average of 2.6 minutes. And the slowest deployment time, in the bottom 5% of the Alexa scoring, was 36.1 minutes.

The DevOps metrics-business success connection is also true when it comes to deployment frequency. The survey showed 75% of companies roll out top projects less than 13 times a week. But for those in the 95th percentile of Alexa ranking, it's a completely different picture. They deploy 32 times a week -- a number that is over 24 times more often than the bottom 5%.

Stability first

But Rose stressed deployment frequency is not the total picture of a healthy DevOps organization. Companies without strong testing in place may end up deploying more often, simply to fix the bugs.

Consider these DevOps metrics: Companies that spend more than 12 minutes on a build deploy a median of 8.3 times a week but also have a much higher percentage of main branch instability (2.7%).

"Once you can get to a firm stable footing, you can branch out to microservices and polyglot development and other things," Rose said. "But you've got to get to stable first."

Dig Deeper on IT operations careers and skills