Market research reveals stubborn gaps between DevOps and IT security teams, as IT experts warn against an over-reliance on short-term measures of effectiveness.
Survey data released this week by two major IT vendors reflect ongoing struggles with burnout and DevSecOps collaboration among IT teams, but some say the problem lies within the data metrics themselves.
A report from Google's DevOps Research and Assessment (DORA) team found that, for the first time since 2014, most respondents to its survey qualified as high performers or elite performers according to its DevOps metrics. These metrics are sorted into four categories: deployment frequency, lead time for changes, time to restore service and change failure rate. A fifth category, which measures operational reliability, was also added for the 2021 "Accelerate State of DevOps" survey, results from which were published this week.
DORA sorts organizations into clusters of elite, high, medium and low performers according to how they measure up in those five categories. Elite performers, for example, deploy software updates on demand multiple times per day with lead time -- the time it takes to go from code committed to code successfully running in production -- less than one hour. Elite performers also take less than an hour to restore services in the event of an outage, and changes they make fail less than 15% of the time.
Though more respondents ranked highly in those categories, only a modest number of survey respondents said they adhere to DORA-defined DevOps security best practices, or DevSecOps. The Google report broke down these best practices and how they're followed among 1,200 respondents to the survey as follows:
58% said they test code for security;
54% said they incorporate security reviews into every phase of development;
60% said they perform security reviews;
49% build code preapproved for security; and
63% invite InfoSec pros into the development process "early and often."
"Around half of our sample actually does [all] of the security best practices that we prescribe in our report," said Dustin Smith, staff user experience researcher at Google. "There are just minor percent improvements [compared with past years] -- nothing significant."
Google report shows DevOps growth
Meanwhile, a survey by Forrester Research commissioned by VMware, which also issued a report on results this week, painted an even bleaker picture of DevSecOps practices among its 1,475 respondents.
When asked if development was involved in security strategy planning, 45.1% of development respondents said they were involved, but only 37.8% of security respondents said they involve development teams, the report stated. Developers are even less involved in security strategy planning than they think they are.
Of the 477 development team managers in the survey, just 22% strongly agreed that their development teams understand which security policies they are expected to comply with. Only 31% of those respondents strongly agreed their development teams know how to appropriately comply with security procedures.
DevSecOps may suffer from misuse of metrics
In the wake of these reports, industry experts said the less-than-encouraging DevSecOps results stem, in part, from a misuse of metrics within enterprises, including the ones DORA uses to evaluate DevOps maturity.
Bryan Finster
"People are using these metrics as goals and causing harm," said Bryan Finster, value stream architect for the U.S. Department of Defense (DoD) Platform One DevSecOps initiative. "I've seen teams have [objectives and key results] given to them with the message, 'We're going to get our deploy frequency to this level by next quarter.' And that's not how it works."
Instead of focusing on the overall goal of delivering value to customers through the software deployment process and optimizing it accordingly, many organizations simply measure software deployment frequency from quarter to quarter, which misses the point, Finster said.
"It's because of incentives. In the private sector, those incentives are quarterly numbers, golden parachutes and the veneer of success," he said. "Where's the long-term planning?"
Daniel Kennedy, an analyst at 451 Research, part of S&P Global, also urged caution among DevOps organizations as they consider metrics, including those used in vendor surveys to evaluate the industry's maturity. For example, Kennedy questioned how realistic some of DORA's security best practices are for mainstream enterprise organizations, such as having security professionals build preapproved code libraries for developers.
Leadership thinks of developers as being a glorified typing pool. If you don't treat professional software engineers like professionals ... you get the quality you deserve.
Bryan FinsterValue stream architect, DoD Platform One
"Coding is one of the least [common] skill sets among security teams," Kennedy said. "That almost sounds like Google taking what works at Google and attributing it to other places."
Burnout adds to DevSecOps malaise
Some 50% of respondents to DORA's survey reported burnout, according to Google's Smith.
Both burnout and poor quality of code from a security perspective stem from similar overemphasis on "developer productivity" without properly aligning business interests with what developers are expected to deliver, in Finster's view.
"Tools are 10% of the problem. The rest of it is us," he said. "We have a whole generation of developers raised to think of themselves as a glorified typing pool, and a lot of leadership thinks of developers as being a glorified typing pool. And if you don't treat professional software engineers like professionals ... you get the quality you deserve."
Google's Smith stopped short of connecting burnout with a lack of adherence to security best practices, but the symptoms of burnout can include finding ways to skip over things, said 451 Research's Kennedy.
"The net effect of burnout is lower productivity, which doesn't necessarily affect security," Kennedy said. "But taking shortcuts can potentially have that effect."
Now what? How to fix DevSecOps
Among the suggestions to improve DevOps security offered by Google's researchers is a correlation it found between good documentation and strong adherence to security best practices.
"High-quality documentation drives the success of a variety of capabilities and security is no exception," the report stated. "We found that teams with high-quality documentation were 3.8 times as likely to integrate security throughout their development process."
But the most important factor in improving the security of code delivered by DevOps teams -- along with every other measure of effective software delivery -- is to tackle a persistent corporate culture that wants to embrace the velocity of DevOps without a change in mindset. This was a sentiment reflected in both Google and VMware's reports, as well as by IT experts such as Finster and Kennedy.
The DORA report's analysis of DevOps culture referenced research by sociologist Ron Westrum, who set forth three broad categories to describe the culture of organizations: pathological, bureaucratic and generative. What Westrum defined as "pathological" culture, marked by low cooperation and scapegoating -- and which DORA identified as an inhibitor to DevOps success -- has been on display in the COVID-19-driven shift to remote work over the last 18 months, Kennedy said.
"The way some companies reacted to remote work, as if it was so novel, tells me they never engaged in any serious business continuity exercises," he said. "And the way people have demanded things [from remote employees], monitored them, required them to be on video all day -- it shows a fundamental lack of trust."
To Finster, he said this shows that a "wall" where developers used to "throw over" code to ops has now shifted into the space between business management and DevOps teams. DevSecOps outcomes won't change until that wall comes down, too, he said.
"People are using the wrong mindset," Finster said. "Software development is not a manufacturing floor ... and it's not about tracking the number of code commits someone makes, but [rather], their business knowledge about the problems you're trying to solve."
Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
Bryan Finster
