DevOps security shifts left, but miles to go to pass hackers

DevOps security tools aren't enough

A cybersecurity tool is only as good as how it's used, Sanabria said, citing the example of a Target breach in 2013, where security software detected potentially malicious activity, but IT staff didn't act on its warnings. In part, this was attributed to alert fatigue, as IT teams increasingly deal with a fire hose of alerts from various monitoring systems. But it also has to do with IT training, Sanabria said.

"In the breach research I've done, generally everyone owned [the tools] they needed to own," he said. "They either didn't know how to use it, hadn't set it up correctly, or they had some kind of process issue where the [tools] did try to stop the attacks or warn them of it, [but] they either didn't see the alert or didn't act on the alert."

The attackers have innovated, and that's one of the problems with our industry -- we're at least five years behind the attackers.

DevOps security, or DevSecOps, teams have locked down many of the technical weak points within infrastructure and app deployment processes, but all too often, the initial attack takes a very human form, such as a spoofed email that seems to come from a company executive, directing the recipient to transfer funds to what turns out to be an attacker's account.

"Often, breaches don't even require hacking," Sanabria said. "It requires understanding of financial processes, who's who in the company and the timing of certain transactions."

Preventing such attacks requires that employees be equally familiar with that information, Sanabria said. That lack of awareness is driving a surge in ransomware attacks, which rely almost entirely on social engineering to hold vital company data hostage.

Collaboration and strategy vital for DevOps security

Thus, in a world of sophisticated technology, the biggest problems remain human, according to experts -- and their solutions are also rooted in organizational dynamics and human collaboration, starting with a more strategic, holistic organizational approach to IT security.

Jeremy Pullen

"Technology people don't think of leadership skills and collaboration as primary job functions," said Jeremy Pullen, CEO of Polodis, a digital transformation consulting firm in Atlanta. "They think the job is day-to-day technical threat remediation, but you can't scale your organization when you have people trying to do it all themselves."

An overreliance on individual security experts within enterprises leads to a 'lamppost effect,' where those individuals overcompensate for risks they're familiar with, but undercompensate in areas they don't understand as well, Pullen said. That kind of team structure also results in the time-honored DevOps bugaboo of siloed responsibilities, which increases security fragility in the same way it dampens application performance and infrastructure resilience.

"Developers and operations may be blind to application security issues, while security tends to focus on physical and infrastructure security, which is most clearly defined in their threat models," Pullen said. "Then it becomes a bit of a game of Whac-a-Mole ... where you're trying to fix one thing and then another thing pops up, and it gets really noisy."

Instead, DevSecOps teams must begin to think of themselves and their individual job functions as nodes in a network rather than layers of a stack, Pullen said, and work to understand how the entire organization fits together.

"Everyone's unclear about what enterprise architecture is," he said. "They stick Jenkins in the middle of a process but might not understand that they need to separate that environment into different domains and understand governance boundaries."

Effective DevOps security requires more team practice

Strategically hardening applications and IT management processes to prevent attacks is important, but organizations must also strategically plan -- and practice -- their response to ongoing security incidents that can and will still happen.

"Cybersecurity so far has been focused on solitary study and being the best technical practitioner you can be, and building stand-alone applications and infrastructure to the best technical standard, which reminds me of golf," said Nick Drage, principal consultant at Path Dependence Ltd., a cybersecurity consulting firm based in the U.K., in a presentation at DevSecCon in Seattle last month. "But in reality, cybersecurity is a fight with an opponent over territory -- much more like American football."

As long as security is practiced by isolated individuals, it will be as effective as taking the football field armed with golf clubs, Drage said. Instead, the approach should be more team-oriented, cooperative, and, especially, emphasize team practice to prepare for 'game time.'

This is the future of governance -- controlling risk on the human side of our systems.

American football defenses are particularly instructive for DevOps security strategy ideas about Defense in depth, Drage said in his presentation. Among other things, they demonstrate that an initial incursion into a team's territory -- yards gained -- does not amount to a breach -- points scored. IT teams should also apply that thinking as they try to anticipate and respond to threats -- how to protect the 'end zone,' so to speak, and not just their half of the field.

Thinkst's Sanabria uses a different analogy -- the DevOps security team as firefighters.

"We're not going to get good at this if we don't practice it," he said. "We buy all the tools, but imagine firefighters if they'd never donned the suits, never driven the truck, never used the hose and they're not expecting the amount of force and it knocks them down. Going out to their first fire would look like a comedy."

And yet that's exactly what happens with many enterprise IT security teams when they must respond to incidents, Sanabria said, in part because companies don't prioritize experiential learning over informational training.

The good news is that IT analysts expect the next wave of DevOps security to look very much like chaos engineering used in many organizations to improve system resiliency, but with a human twist. Organizations have begun to emerge such as OpenSOC, which sets up training workshops, including simulated ransomware attacks, for companies to practice security incident response. Companies can also do this internally by treating penetration tests as real attacks, otherwise known as red teaming. Free and open source tools such as Infection Monkey from Guardicore Labs also simulate attack scenarios.

Charles Betz

Tech companies such as such as Google already practice their own form of human-based chaos testing, where employees are selected at random for a 'staycation,' directed to take a minimum of one hour to answer work emails, or to intentionally give wrong answers to questions, to test the resiliency of the rest of the organization.

"Despite the implications of the word 'chaos,' some companies are already presenting chaos engineering to their risk management leaders and auditors," said Charles Betz, analyst at Forrester Research. "This is the future of governance -- controlling risk on the human side of our systems."