To secure DevOps, break culture and tooling barriers

After majoring in infosec and risk management in college and spending 10 years in the infosec field, one thing became extremely clear to Julien Vehent: The industry's approach to security was slow and disconnected from the modernization of operations. Then, as DevOps came into the picture and industrialized the way cloud and web services were created, deployed and managed, he realized security simply wasn't transforming at the same speed.

So, what did Vehent do when faced with the challenge of changing the way security is approached in a DevOps world? He wrote about it.

Securing DevOps: Security in the Cloud, published by Manning Publications, is the result of Vehent's research. Here, he discusses the book and how to sandwich the sec in DevSecOps to achieve a secure DevOps initiative, as well as why his target reader didn't turn out to be the only constituency finding value from his book.

Editor's note: This interview has been lightly edited for length and clarity.

What needs to change to effectively secure DevOps? Is it on the DevOps side or the security side?

Julien Vehent: Both. The DevOps industry has to grow and mature to adopt security, and the security industry has to understand how DevOps changes the way we manage infrastructure to implement security into it.

How do we get DevOps teams to adopt security and integrate security in their day to day? To answer that, we have to look at how DevOps changes infrastructure and operations. Every time you bring in a server, you have a human being involved with that task -- the human places trust into that server by setting it up. With cloud infrastructure, this model doesn't apply anymore. Systems are brought up into the infrastructure automatically as the load increases.

How can that needed trust be achieved?

Vehent: For the longest time, we've approached security from a compliance angle. What the DevOps culture brought in is how we build security inside DevOps teams -- not as compliance at the end.

This is often called shift left. Historically, if you look at how developers brought services to production, security was done at the end. Shifting left brings it closer to the design phase, at the very beginning. We want to build that culture into the teams building and running the services as early as possible. We do this by embedding security engineers into development and operations teams. They still report to their security team, but they basically belong to the other team -- they adopt their tooling and their culture, and in the bringing the security knowledge, they inject security into the core of the operations and the development cycles. That's basically what DevSecOps is about: You put security in the middle of dev and ops, not at the end as a compliance step.

Beyond the culture shift, is there a shift in tooling and security mechanisms?

Vehent: Probably 90% of the tooling we used 10 years ago is gone. We had to reinvent everything, and we continue to reinvent everything. Some of it will come back. Intrusion detection systems were gone from most cloud infrastructure, but they're starting to come back as those infrastructures mature.

Secrets management is a good example where you see new tools -- Mozilla wrote one, called SOPS. There are a number of them out there, including HashiCorp Vault, plus AWS, GCP [Google Cloud Platform] and Azure bring their own. These tools manage secrets of the infrastructure in a fundamentally new way, leveraging the trust models of cloud infrastructures to better protect the secrets of the environment. These tools simply didn't exist even five years ago.

What this means is that a lot of security teams have shifted to being software engineering teams. They're still security teams and have a security specialization, but they do a lot of software engineering. They engineer the security tools, and they engineer the security automation the infrastructure is going to use. In comparison, even just 10 years ago, security teams were mostly composed of network engineers or security compliance engineers. With DevSecOps, programming your security tools and your automation is key to succeeding.

What are teams finding most difficult to secure DevOps?

Vehent: In security teams, there's a feeling that they're reducing security by adopting DevOps -- that most of the tooling and knowledge is going away, and therefore, security would decrease. A lot of teams end up pushing back because they don't want to put infrastructure at risk.

At an executive leadership level, organizations need to train their people to break those barriers and inject security at all phases of developing, running and operating cloud services. This has to come from the top.

To overcome this, we must retrain security teams to better understand and leverage the security that exists in cloud environments. Second, we need to change the perception that the culture of the organization may not be receptive to security teams being so closely embedded with dev and ops. A lot of organizations still expect, at the executive level, that security will be at the end of the process -- essentially, compliance. This often trickles down to the engineering teams themselves, and they may not be ready to accept security engineers getting embedded into their day-to-day work. At an executive leadership level, organizations need to train their people to break those barriers and inject security at all phases of developing, running and operating cloud services. This has to come from the top.

The biggest challenge I hear from people who are not successful at implementing secure DevOps is that they're not in a position to reach out to their dev and their ops -- they end up doing it out of compliance because they can't do it earlier in the process.

In the book, you wrote that it translates some of your real-world experiences integrating security into DevOps. Is there a specific example?

Vehent: When I started at Mozilla, I was working on this security project called Mozilla InvestiGator. It was an endpoint security project we were developing in-house to interrogate our servers. One thing that became clear is the way we were implementing it wasn't about what integrated into the operating infrastructure -- we were trying to shove a security tool on top of the infrastructure without integrating it into the existing tooling. We were pretty successful with the project. But I always had the impression that we could integrate our security tooling into the infrastructure better.

When I started working on AWS and the cloud infrastructure of Mozilla, a lot of the security needs we had were, in fact, already solved by tooling or techniques the operations teams were using. For example, we wanted an inventory of all the systems -- well, they already had that because they needed one to do their operations. So, instead of building security tools to solve security problems, I began to think we could actually grow operational tools and get our security answers from those tools. This is really when I started to look at how to integrate security directly into operations and development.

Who would you say is the target reader for the book?

Vehent: That's an interesting question because I have a picture of who the target reader is in my mind, but I've also heard from a lot of people who don't fit the template at all and have found value from the book.

My target reader is typically either a junior security engineer who wants to break into the DevSecOps field or a more senior engineer outside the security field -- maybe a developer or the sys admin who wants to move into security. That's why the book spends a lot of time on more advanced security concepts and less on basic computer science skills. I assume the readers have some scripting and operations experience.

But I've also heard a lot of senior security folks -- sometimes with decades of experience -- found value from the book because they've never worked in a DevSecOps environment. They hear the buzzwords, and their management is telling them they have to be agile, they have to adopt DevOps. But they can't really put their finger on it -- what does it mean? How will my day to day change? The book does a pretty good job explaining what the day to day of a DevSecOps engineer will look like and what you should invest in for the couple of years and beyond.

About the author

Julien Vehent

Julien Vehent is the leader of security architecture for Mozilla's Cloud Services division. He is responsible for defining, implementing and operating the security of web services that millions of Firefox users interact with daily. Vehent consults with development and operations teams on risks and security and helps integrate controls in their build pipeline.

Vehent has been focusing on developing, operating and securing internet services for the past 15 years, starting as a Linux sys admin and graduating with a master's degree in information security in 2007. He gained experience in the financial sector in France, working on the customer portals of various French banks, then in startups in the U.S. before joining Mozilla in 2013.

He is a programmer and author of security tools, started in C and released Honeybrid, a honeypot proxy, back in 2007. In 2012, Vehent wrote AFW, an automated firewall that generates host-based rules within a Chef managed environment. At Mozilla, he wrote Cipherscan (TLS auditing), Mozilla InvestiGator (distributed digital forensics), SOPS (secrets encryption), Userplex (identity management) and many smaller tools to automate security in the organization.

Vehent also studied the Kaizen approach and observed Agile transform into DevOps and Kanban. The benefits of these methods on the productivity of both chemical and software industries forged his interest in evolving the information security community toward faster improvement cycles.