DevSecOps strategy mimics cloud shared responsibility model
Real-world DevSecOps centralizes IT security around the CI/CD pipeline but defines separate roles that each IT constituency plays in securing apps and infrastructure.
Enterprises that have successfully established DevSecOps use various CI/CD and automation tools but take an approach to team organization that echoes the shared responsibility model employed by hyperscale cloud providers.
As with DevOps itself, the initial focus for most mainstream IT pros as DevSecOps became a popular term two years ago was on which security automation tools to include in DevOps pipelines and how to explain what those tools do to auditors. By last year, however, experts urged a shift from tools to strategy, collaboration and communication, in order to establish true DevSecOps.
For many organizations, these issues are nebulous and difficult to tackle, but early DevSecOps adopters point the way toward organizational best practices based on shared responsibility. Cloud providers such as AWS split infrastructure security responsibilities at the operating system level, securing the lower layers of the infrastructure, while leaving the upper layers to customers.
Similarly, DevSecOps organizations delegate specific, manageable areas of responsibility to developers, security and IT ops pros, respectively, within a set of shared tools. Some even emulate the customer-provider relationship among teams as well.
"The developers are essentially our customers [on the platform team], and the actual Mettle customers are really the developers' customers," said Steven Wade, platform lead at Mettle, a fintech company in the U.K. "As long as we can keep them happy, and we get out of their way and allow them to be able to innovate, we've succeeded."
Shared responsibility, but separate focus points
Security automation has been part of application delivery at Mettle since the company was founded in March 2017. Developers have assumed responsibility for writing secure code, while security and ops engineers on Wade's platform team handle security for AWS cloud infrastructure components, including Kubernetes and containers.
Steven Wade
Mettle bases its security philosophy on GitOps -- it segments Kubernetes clusters and code repositories among multiple Weaveworks Flux controllers, which automatically apply code changes committed to master branches to the live IT environment.
This means the company's IT infrastructure can be deployed and deleted very quickly and consistently -- "Twenty-five to 27 minutes," Wade said. "You can tell I've timed it."
That infrastructure benefits, security-wise, from being built on ephemeral resources that are never upgraded or patched, a setup known as immutable infrastructure.
GitOps' focus on code repositories also helped establish the shared responsibility model for DevSecOps at Mettle, Wade said. The whole IT team works together collaboratively within Git, but each employee has a specific focus and owns a specific part of the code.
"Every [Git] repository has a set of code owners," Wade said. "The code owners are people that understand the changes that are being made, and they're the best people to make judgments on them."
Every [Git] repository has a set of code owners ... people that understand the changes that are being made, and they're the best people to make judgments on them.
Steven WadePlatform lead, Mettle
For example, monitoring and managing Kubernetes secrets within Git repos falls to security engineers. Mettle's platform operations engineers maintain Kubernetes resources in production using Fairwinds' Polaris to identify deployment configuration errors, and Gatekeeper, an open source Kubernetes admission controller that enforces access control rules.
Meanwhile, developers ensure application security through a set of continuous integration tests pre-deployment, using open source tools such as Aqua's Trivy vulnerability scanner, Conftest for configuration testing and kubeval, which scans Kubernetes YAML and JSON files.
The platform team has also prepared a set of Helm charts that developers can use to ensure that their microservices tie in with infrastructure smoothly, including for security purposes, as they're deployed through Flux.
"Developers aren't Kubernetes experts, nor should they have to be -- they're trying to ship features," Wade said. "We do a back-end job every microservice uses with all the best practices already embedded ... [developers] just have toggles and they [only] have to understand the [Helm] values files to onboard their new microservice."
Informatica: Different tools, same techniques
While Mettle prefers to make its IT environment elusive to would-be attackers through ephemeral infrastructure and DevOps automation, data management vendor Informatica relies on security scans and infrastructure security monitoring through tools such as StackRox and Elasticsearch. But it, too, has organized DevSecOps teams around the shared responsibility model.
Informatica began to realign its IT engineers into a DevSecOps model when it migrated its IT infrastructure to cloud in 2017. It later deployed StackRox after shifting from VMs to Kubernetes and containers in 2018.
Pathik Patel
StackRox handles multiple aspects of security automation at Informatica, but as with Mettle's delegated responsibilities within shared code repositories, each engineering team at Informatica uses separate parts of that overall framework. Developers scan container images with StackRox before deployment, while ops engineers report security issues flagged by the tool's runtime security monitoring to the security team and developers. The security team uses StackRox as a repository for security policies.
"This is an extension of how the AWS shared responsibility model plays out," said Pathik Patel, head of cloud security for the company. "We as security engineers are responsible for the writing policy and monitoring it -- when we come across a security issue, we can issue a patch, but ultimately it needs to be reported back to developers to fix it."
The initial adjustment to the DevSecOps model required a change in mindset for developers, but building in vulnerability detection ahead of app deployments ultimately lowered their stress around security, Patel said.
"In the first year, we were reporting all these vulnerabilities and issues, and the numbers were piling up," he said. "[Developers said], 'You always bring this up at the last minute' ... we had to fix their process."
Steven Wade
Pathik Patel
