DevSecOps veterans share security strategy, lessons learned

SEATTLE -- DevSecOps strategy is as much an art as a science, but experienced practitioners have a few pointers about how to approach it, including what not to do.

The first task DevSecOps newcomers should undertake, according to Julien Vehent, security engineer at web software firm Mozilla, is to design an effective IT security team structure. In his view, the ideal is an org chart that embeds security engineers with DevOps teams but has them report to a centralized security department. This structure helps to balance their impact on day-to-day operations with maintaining a cohesive set of broad goals for the organization.

This embedding can and should go both ways, Vehent added -- security champions from DevOps teams should also have access to the central security organization to inform their work.

"Sys admins are great at security," he said in a presentation here at DevSecCon this week. "Talk to people who do red teaming in their organization, and half the time they get caught by the sys admins."

Once DevOps and IT security teams are aligned, the most important groundwork for improved DevOps security is to gather accurate data on IT assets and the IT environment, and give IT teams access to relevant data in context, practitioners said.

"What you really want from [DevSecOps] models is to avoid making assumptions and to test those assumptions, because assumptions lead to vulnerability," Vehent said, recalling an incident at Mozilla where an assumption about SSL certificate expiration data brought down Mozilla's add-ons service at launch.

Since then, Vehent's mantra has been, "Avoid assumptions, trust the data."

Effective DevSecOps tools help make data-driven decisions

Assumptions lead to vulnerability. Avoid assumptions, trust the data.

Once a strategy is in place, it's time to evaluate tools for security automation and visibility. Context is key in security monitoring, said Erkang Zheng, chief information security officer at LifeOmic Security, a healthcare software company, which also markets its internally developed security visibility tools as JupiterOne.

"Attackers think in graphs, defenders think in lists, and that's how attackers win," Zheng said during a presentation here. "Stop thinking in lists and tables, and start thinking in entities and relationships."

For example, it's not enough to know how many AWS Elastic Cloud Compute instances an organization has, but to understand their context by analyzing multiple factors, such as which ones are exposed to the internet, both directly and through cross-account access methods.

IT pros can configure such security visibility graphs with APIs and graphing databases, or use prepackaged tools. There are also open source tools available to help developers assess the security of their own applications, such as Mozilla's Observatory.

LifeOmic also takes a code-driven, systematized approach to DevOps security documentation, Zheng said. Team members create "microdocuments," similar to microservices, and check them into GitHub as version-controlled JSON and YAML files.

Another speaker urged IT pros new to DevSecOps to take a templatized approach to IT training documentation for cybersecurity that explains, in jargon-free language, the reasons for best practices, and give specific examples of how developers often want to do things, versus how they should do things to ensure application security.

"The important thing is to make the secure way the easy way to do things for developers," said Morgan Roman, application penetration tester at electronic signature software maker DocuSign. "Find bad patterns, and make the safer way to do things the default."

DevSecOps how tos -- and how NOT to dos

Attackers think in graphs, defenders think in lists, and that's how attackers win.

Strategic planning and assessments are important, but certain lessons about DevOps security can only be learned through experience. A panel of cybersecurity practitioners from blue-chip software companies shared their lessons learned, along with tips to help attendees avoid learning the hard way.

Multiple panelists said they struggled to get effective results from code analysis tools and spent time setting up software that returned very little value or, worse, created false-positive security alerts.

"We tried to do things like, 'Hey, let's make sure that we aren't just checking in secrets to the code repository,'" said Hongyi Hu, security engineer at SaaS file-sharing firm Dropbox, based in San Francisco. "It turns out that there's not really a standardized way of doing these things. … You can find things that look like secrets, but secrets don't always look like secrets -- a really weak secret might not be captured by a tool."

Ultimately, no tool can replace effective communication within DevOps teams to improve IT security, panelists said. It sounds like a truism, but it represents a major shift in the way cybersecurity teams work, from being naysayers to acting as a consulting resource to apps teams. Often, a light-handed approach is best.

"The most effective strategy we got to with threat modeling was throwing away any heavyweight process we had," said Zane Lackey, co-founder and CSO at WAF vendor Signal Sciences in Los Angeles. "As product teams were developing new features, someone from the security team would just sit in the back of their meeting and ask them, 'How would you attack this?' and then shut up."

It takes time to gain DevOps teams' trust after years of adversarial relationships with security teams, panelists said, but when all else fails, security pros can catch more flies with honey -- or candy.

"We put out bowls of candy in the security team's area and it encouraged people to come and ask them questions," Lackey said. "It was actually wildly successful."