DevOps security takes on the dark side of digital transformation

The DevOps security ideal

Jeremy Pullen

IT ops pros may see security as a secondary goal to DevOps, or resist DevOps concepts that contradict ITIL security and other IT service management best practices. But both of those mindsets are dangerously wrong, said Jeremy Pullen, CEO and principal consultant at Polodis Inc., a DevSecOps and Lean management advisory firm in Tucker, Ga., that works with large enterprise clients.

"IT pros that work in ticketing and configuration databases create a separate set of data from version-controlled code and think they're following best practices," Pullen said. "But they're actually following institutionalized incompetence that's stuck in the dark ages."

DevOps security is the only viable approach as digital assets become crucial to the enterprise bottom line, Pullen said. Ideally, IT employees should access enterprise production environments only with developers' version-controlled code, checked in to an automated delivery system -- a setup that limits internal security threats, he said. The DevOps practice of small, iterative changes to modular infrastructure also reduces the attack surface of IT systems for outside threats.

However, DevOps proponents are mistaken to emphasize the gatekeeper mentality that relies on human approvals or manual work to deploy production application changes, Pullen said.

"Automated changes to production scares IT folks, but version control should be the gatekeeper," he said. "Version-control systems are fully auditable, reproducible and traceable."

Industries sound alarms for DevOps security

Regulated businesses often cite auditors as the main reason for the gatekeeper approach to production application deployments, because they often don't understand DevOps and the changes IT pros want to make.

However, legislators, policymakers, and the regulatory and risk management industries are increasingly aware of the market disruption risk tied to IT security, and public policy in the last year reflects a better grasp of cybersecurity. The European Union's General Data Protection Regulation, for example, specifies a goal to ensure customer digital privacy, rather than a technical method to attain that goal. In the U.S., the Office of the Comptroller of the Currency has started to regulate FinTech companies, even if those companies don't qualify as banks under the OCC's traditional purview.

The fintech industry has fought this regulation, but some IT security experts believe government policies will drive DevOps security best practices. They cite a pending Ohio law that indemnifies companies against liability for data breaches, provided they comply with certain cybersecurity frameworks.

"[Ohio's law] shows a thousand-person company with five IT people why they should care about cybersecurity," because it provides a direct means to avoid the potential costs of data-breach lawsuits, said Ron Gula, former CEO of Tenable and president of Gula Tech Adventures, a venture capital firm that invests in IT security startups.

Meanwhile, risk management firms now embrace the collaborative DevOps organizational style.

"Conversations have begun to shift away from the financial crisis [of 2008], which prompted focus on efficiency of capital and liquidity and financial institutions, to reassess risk management in cybersecurity," said Marc Saidenberg, co-leader of London-based Ernst & Young's Global Regulatory Network. "It's part of the dialogue regulated firms have to have with their auditors."

IT wrestles with DevSecOps disconnect

Identify everything you do that touches production and determine how you're going to eliminate that access. Assume that five years from now you won't have direct access to production systems, and think about how that will change your job.

Enterprises that have tried to educate auditors on DevOps security report mixed results, however.

"I don't think the audit world has caught up to what we're doing," said Julie Chickillo, vice president of security at Beeline, a company headquartered in Jacksonville, Fla., which markets workforce management and vendor management software.

The effort to enlighten auditors meant Chickillo and her team spent about 20% more time on audits in the company's first year of DevOps, she said in a presentation at DevSecCon in Boston this month. But there are also bright spots: In at least one instance, an auditor offered Chickillo valuable insight into DevSecOps best practices and how to document automated security checks.

"No findings [from a testing tool] is not the same as proof that [a check] ran," Chickillo said. "That's the important question: How do you know?"

Alex Bekker

It's unlikely regulators will get ahead of IT security breaches, HackerOne's Bekker said, but he hopes "it will make dollars and sense" for companies to invest in DevOps security without regulatory intervention. Top levels of enterprise management must impose new security requirements on IT vendors, which already happens more often than in years past, Bekker said.

Beeline has begun to change the way it vets IT security vendors, and now its DevOps engineers evaluate IT security products before its security team looks at them. But this practice has also illuminated a gap in the IT security market, Chickillo said.

"Traditional security vendors that are trying to get into [emerging application deployment methods of] containers and microservices don't seem to understand the purpose of them," she said. "Companies that started out in the container environment have good technology, but [their products are] just out of beta."

Next steps in DevOps security for IT ops pros

So, how do IT operations professionals in the trenches push DevOps security forward? For the long term, Pullen suggested a fourfold strategy, with different approaches to greenfield, brownfield, custom and off-the-shelf applications. The more legacy and vendor-controlled the technology, the more ongoing audits and upgrades are required. With greenfield and custom apps, there are more opportunities to "do it right the first time," he said.

For the short term, Pullen has two primary recommendations.

"One, identify everything you do that touches production and determine how you're going to eliminate that access," he said. "And, two, assume that five years from now you won't have direct access to production systems, and think about how that will change your job."