Security automation charges IT ops to build the 'paved road'
As IT ops pros evolve into SREs, they play a key role in security automation and create secure paths to production deployment for application developers.
As application release velocity and cloud-native application complexity increase, security automation helps keep control over risk -- and IT ops pros play a key role in laying that foundation.
Over the last three years, as DevOps emerged, many IT organizations struggled to automate application deployments to production without security "gates," or points of manual approval for compliance, which slowed down software releases. Organizations were reluctant to trust automated tools to deliver apps reliably to production infrastructure. In response to these bottlenecks, developers would sometimes circumvent secure processes, and while code security and quality are key tenets of the original DevOps philosophy, they were abandoned in favor of moving fast. Power struggles also sometimes ensued between IT security teams that were inexperienced in Agile and DevOps methodology and developers under pressure to continuously produce code when it came to selecting IT security tools.
Little by little, however, that picture has begun to change. IT automation tools for infrastructure as code, configuration management and security testing now build secure practices into the DevOps process. Many mainstream enterprise organizations are working toward a GitOps model of application delivery, in which humans perform little to no manual intervention in application deployments -- an approach experts have long argued is more secure, because it reduces the risk of human error and tampering.
"[The GitOps model] is pretty much standard practice in a lot of places," said Jeremy Pullen, CEO of Polodis, a digital transformation consulting firm in Atlanta.
One of the big shifts I've seen in the industry as a whole is from a focus on finding bugs to [creating] secure-by-default environments where ... it's really hard to do the wrong thing.
Clint GiblerResearch director, NCC Group
For security purposes, this creates what's become known as the "paved road" to production, a term coined by Netflix engineers. Developers still have full flexibility to deploy applications as they choose, but the path of least resistance is the most secure. At organizations that take that approach, IT ops pros are the builders of that secure path, through automated CI/CD pipelines, configuration management and infrastructure as code tools that enforce good practices, eliminate manual errors and configuration drift.
"One of the big shifts I've seen in the industry as a whole is from a focus on finding bugs to [creating] secure-by-default environments where, if you use the frameworks and processes that are standard, it's really hard to do the wrong thing," said Clint Gibler, research director at NCC Group, an IT security consulting firm in San Francisco. "There's a strong focus on security basics, where developers don't have to think about security as much, and that makes them more productive. That's one of the biggest, most important shifts I've seen."
Security automation reduces manual effort, errors
For traditional enterprises, full collaboration from the start of the product planning process among application developers, IT security teams and IT ops remains a work in progress. But security automation tools that scan code and containers, automate network microsegmentation and provision infrastructure as code have freed IT teams to focus on the next phase of maturity.
"We're a financial organization with really tight controls around security, but we wanted to move quite quickly, and what we became conscious of was that we wanted to segment workflow based on their risk profile," rather than shore up traditional perimeter-focused security in the IT infrastructure, said Colin Lennox, head of technology and service delivery at Baillie Gifford, an investment management firm in Edinburgh, U.K. "To do that through traditional hardware-based segmentation would have been very expensive and cumbersome to manage."
Baillie Gifford began to experiment with a software-based network microsegmentation tool from Illumio two years ago, and recently put it into production for VM-based workloads. Next, it will use Illumio to automatically secure container workloads, as it adopts public cloud services, Docker and Kubernetes. The company has yet to build an end-to-end CI/CD pipeline, but it uses Twistlock's container security tools to automatically scan containers before they're deployed in development and staging environments, which must meet the same security standards as production. The company has also decided to standardize infrastructure as code and configuration management on Puppet, in part to minimize security risks, Lennox said.
"People are the biggest risk -- you can set policy, but there's always the risk [of errors]," he said. "Puppet for patching and infrastructure as code allow us to get away from doing things on a manual basis and refocus on responding more quickly to immediate vulnerabilities."
Tools such as Puppet and Chef have been used for years already at some traditional enterprises, but advances in their support for security automation and heightened awareness of DevOps security best practices mean their use has changed within ops teams.
Brittany Woods
"With Chef, we make sure that we've empowered all of our infrastructure users and developers to be able to write their own cookbooks," said Brittany Woods, an automation engineer at automotive data service provider Carfax Inc. in Centreville, Va. "We have pipelines for cookbook and code delivery to ensure they meet our standards, and we'll add Chef InSpec for [security and compliance] scanning."
InSpec is a highly flexible policy-as-code tool that has a number of potential security automation applications, and Woods' team at Carfax hopes to put them in place companywide to shore up DevOps security. Already, InSpec has helped to fully automate system patching and track that systems are kept up to date for auditors. Woods said she'd like to integrate InSpec with the company's ServiceNow IT ticketing system to further root out human error in security incident response.
"As a company and an industry, we're focusing more on automation and DevSecOps, and part of that is making sure all of our systems make certain requirements," Woods said. "This is also a relatively easy tool to use."
Application security monitoring further improves DevOps security
IT ops pros and SREs at security-conscious companies have built security into their IT monitoring and observability practices, which gives organizations a better idea of how to strategically optimize security automation.
Before IT security and SRE teams sat down a little over a year ago to develop a security automation strategy at ActiveCampaign, an email marketing firm in Chicago, they scanned the existing app delivery pipeline and infrastructure with Threat Stack to identify vulnerabilities and areas for improvement.
Chaim Mazal
"I used it as a way to say, 'Hey, we're not following best practices in the way we're building out infrastructure,'" said Chaim Mazal, director of global information security at ActiveCampaign. "We need to tighten up, not only from an infrastructure security perspective, but also from an automation and deployment perspective, and even from a general Linux administration perspective."
Hard data from a technical tool made that an easier conversation, Mazal said, "Instead of, 'Oh, security says we're doing this wrong.'"
Threat Stack data was also the catalyst for the company to deploy a sophisticated identity and access management toolchain made up of Okta identity management, LDAP access control and HashiCorp's Vault for secrets management.
"We got rid of passwords and set up limited temporary access to resources," Mazal said. "We also eliminated nonscalable, ad hoc practices that were past the point of diminishing returns, such as having EC2 users perform highly critical functions across the infrastructure."
Security is an equal member of our engineering team, and security is a clear driver of value for us as a company.
Chaim MazalDirector of global information security, ActiveCampaign
ActiveCampaign SREs also began to deploy repeatable, auditable infrastructure resources with Terraform infrastructure as code, and sped up its response to critical security incidents by attaching SLAs to certain incident types and handling them through a separate Kanban process, rather than adding them to the main development backlog for later sprints.
Security automation has also enabled ActiveCampaign to develop customer-facing features that help it compete with other email marketing and marketing automation companies, such as multi-factor authentication at no extra cost, as well as security analytics for customer accounts.
"Security is an equal member of our engineering team, and security is a clear driver of value for us as a company," Mazal said. "We can't be the No. 1 sales and marketing automation platform without being trusted by our customers."
Dig Deeper on Systems automation and orchestration
Brittany Woods
Chaim Mazal
