Part of:Getting started on a DevSecOps career path
Case study: Scaling DevSecOps at Comcast
Comcast's DevSecOps transformation started small but quickly gained steam, resulting in 85% fewer security incidents in production. Learn more in this case study.
Noopur Davis and Larry Maccherone have been preaching the virtues of built-in security since their days leading a government-funded research initiative at Carnegie Mellon's Software Engineering Institute -- in all, more than two decades.
"The whole idea was to empower software engineering teams to take ownership of security -- build it in, don't bolt it on," said Maccherone, now the DevSecOps transformation lead at Contrast Security. The problem: The program was ahead of its time. "We failed to get it off the ground."
Davis and Maccherone went their separate ways. Then, in 2016, they got the chance to resurrect their old ideas under a new name: DevSecOps. Davis, by then the CISO at Comcast, reached out to Maccherone, who had become heavily involved with Agile and DevOps.
"Noopur said, 'Hey, Larry, DevOps is making it possible for us to actually do what we wanted to do 15 years ago,'" Maccherone said. "'Come to Comcast, and let's try again.'"
Piloting a DevSecOps transformation
Starting with a staff of just 16, Maccherone launched a small DevSecOps pilot program at Comcast. Out of the telecom conglomerate's 600 application development teams, he identified around 10 already practicing what he considered true DevOps, making them ideal candidates for a DevSecOps transformation.
DevSecOps is just DevOps done right.
Larry MaccheroneDevSecOps transformation lead, Contrast Security
"By my definition, DevSecOps essentially means that empowered engineering teams take ownership of their products all the way to production," Maccherone said. "It's really the same definition I have for DevOps. To me, DevSecOps is just DevOps done right."
Before Maccherone's arrival, however, even Comcast's most mature DevOps practitioners were supposed to hand off their software to the company's siloed application security team, which would then "bolt on" security.
"Security would send it back to the developers weeks or, sometimes, even months later," Maccherone said. The process interrupted programmers' flow and undermined DevOps' effectiveness, he added.
"We basically said, 'Here's tooling that works the way developers work and thinks the way developers think, that plugs into your pipeline and provides the feedback directly to you,'" Maccherone said.
Bringing in DevSecOps coaches
Maccherone rolled out the new tools to each participating development team in a 90-minute introductory workshop. He then assigned each group a DevSecOps coach, who would help the developers choose a few core practices to adopt over the next three months -- for example, installing a software composition analysis (SCA) tool in the CI/CD pipeline and scanning for critical vulnerabilities.
The tooling would typically flag a handful of high-priority findings that the team could resolve over the course of a development sprint or two, with extra help available if necessary. "As your coach, I can always bring in an expert in slam-dunking or free throws to get you over a training hump," Maccherone said.
After working its way through the critical alerts, the development team could then adjust the policy dial to not just scan code, but also block it from merging unless clean. "Critical SCA findings accounted for roughly 35% of all security incidents at Comcast," Maccherone said. "Turn the dial to 'blocking,' and you'll never have another one of those get into production ever again."
Over time, a DevSecOps coach would encourage developers to slowly turn up the heat, perhaps also scanning for high- and medium-severity vulnerabilities, for example, or adding interactive application security testing or static application security testing findings to the mix. "The coach's job was not to call your baby ugly, and it was never to tell you that you were doing it wrong," Maccherone said. "It was to say, 'What's the next opportunity to improve?'"
With regular coaching and hourlong workshops every 90 days, a typical DevOps team at Comcast could typically reach DevSecOps maturity in about a year and a half, he added.
A DevOps litmus test
With early efforts going well, Comcast's DevSecOps pilot quickly grew. But an early challenge, according to Maccherone, was convincing company leadership that some development groups weren't ready to participate. Putting the cart before the horse -- introducing DevSecOps to a team that hadn't yet nailed DevOps -- would just cause frustration and waste everyone's time, he argued.
To vet prospective program participants, Maccherone asked developers one key question: whether they trusted the automated tests in their pipeline to invalidate a bad artifact and keep it from getting into production.
"Can you blindly make a change and push it through your pipeline and be confident it isn't going to break?" Maccherone said. "If you can't do that, you're not doing DevOps."
Practicing DevSecOps at scale
Soon, about 100 of Comcast's software development teams were practicing DevSecOps. The results were compelling, with those groups seeing 85% fewer security incidents in production than their legacy counterparts.
Maccherone designed Comcast's small DevSecOps transformation program to scale across its 600 development teams. Because each participating team had just one formal training workshop every few months -- with the rest of the coaching happening on an ad hoc basis -- a single, full-time DevSecOps coach could juggle up to 100 teams per quarter.
Importantly, Maccherone designed Comcast's DevSecOps program to scale, with his dedicated coaches able to work with up to 100 development teams per quarter. To further increase scalability, Comcast also created a federated coaching program, in which someone from outside the DevSecOps pilot team -- say, a security specialist from a standalone business unit -- could train to be a DevSecOps coach.
"They had to use our framework and our tooling," Maccherone said. "And they had to shadow us three times, and then we reverse-shadowed them two times." If they passed, the federated coaches could then lead DevSecOps workshops in their own business units.
Within five years, about half of Comcast's 600-odd development teams had joined the DevSecOps transformation program. At that point, the company decided to transition the remaining teams and shut down its traditional AppSec program. Instead of a siloed team of 400 AppSec specialists, the company would have 100 DevSecOps pros.
"That essentially solved Comcast's cybersecurity hiring problem," Maccherone said. "We were able to do 85% better risk reduction with a quarter of the staff."
