kras99 - stock.adobe.com
How CI/CD pipelines are putting enterprise networks at risk
At Black Hat USA 2022, NCC Group researchers demonstrated how threat actors can compromise CI/CD pipelines and break out into enterprise networks and cloud environments.
LAS VEGAS -- When it comes to the software supply chain, organizations should be concerned about more than just external threats like poisoned open source dependencies.
During a Black Hat USA 2022 session Wednesday, cybersecurity vendor NCC Group presented its findings from studying continuous integration (CI) and continuous delivery (CD) software pipelines at many organizations during the past five years. Enterprises have embraced CI/CD pipelines in recent years to increase speed and efficiency for software development.
While supply chain security has become a concern for enterprises amid growing reports of poisoned software libraries and compromised repositories, NCC Group researchers argued that the pipelines themselves pose significant risk to organizations.
During the Black Hat presentation, NCC Group's Iain Smart, principal security consultant, and Viktor Gazdag, managing security consultant, explained how they were able to compromise virtually every CI/CD pipeline they've tested in the last five years, with some rare exceptions. They described how gaining unauthorized access to pipelines, which are often not properly segmented and hardened, could allow threat actors to turn them into "remote code execution (RCE) as a service" threats that deliver signed malware instead of enterprise applications.
In an interview with SearchSecurity, Smart said CI/CD pipelines are a dangerous attack surface that has gone "unrealized" for a long time. But that might not last, as threat actors continue to probe for weak spots in enterprise environments, particularly around the software supply chain.
"I think a lot of this comes down to a lack of threat modeling, which is a formal way of saying you're just not thinking about what you're deploying," he said.
CI/CD pipeline attacks
Smart and Gazdag detailed several customer engagements where they were tasked with compromising a CI/CD pipeline and then breaking out of the pipeline to gain access to a company's primary infrastructure.
Smart described one engagement where the client granted NCC Group researchers a set of developer credentials, which lacked admin privileges, to see if they could break out of the pipeline. While they had access to just a single Bitbucket repository, they were also able to modify which dependencies were required for the pipeline to build an application.
Smart's team added its own malicious dependency to the mix, and when the pipeline connected to it, NCC Group was granted access to a Jenkins runner. While the runner had a limited environment, Smart said NCC Group discovered a private SSH key lurking in it for unknown reasons. With the key, the researchers were able to gain access to all the organization's Jenkins control nodes and obtain all secrets -- which NCC Group described as any nonpublic data, such as credentials and API keys -- across all the customer's Jenkins projects.
"In this case, it turned out we had full access to deploy anything we liked into the customer's production workload," including malware, Smart said.
In another red team engagement, NCC Group compromised a customer that Smart said was a "relatively hard target." The researchers weren't able to get into the customer's CI/CD pipelines through their usual methods, but they successfully phished a developer and used the developer's credentials to access a wide range of repositories.
Even though the customer had hardened its development environment and limited permissions and access, the red team was able to modify some of the pipelines' functions and grab secrets that had not been filtered out. Eventually, the researchers found a single pipeline that had domain admin access, which gave the red team significant control.
"This shouldn't need to be said in 2022 -- don't run everything as domain admin, please," Smart said. "It's just not a good idea."
Gazdag and Smart discussed other customer engagements where the research team was able to break out of a compromised pipeline into not just the full product environment, but also the customer's cloud and on-premises infrastructure. In one case involving a particularly well-defended customer with extremely limited developer access, Smart was still able to obtain the production API keys and modify the company's AWS environment.
Smart said the customer was initially in denial that such an attack was possible and was so confident that it allowed him to execute the attack. "Two redacted screenshots later, we were on an incident response call," he said.
He also said that code signing software in production is a good practice, but it's not a silver bullet to prevent malicious activity. In fact, it could be used by threat actors to create an RCE as a service.
"A lot of people think, 'Well, I sign everything at the end of my pipeline, so I'm secure,' and they're kind of correct," Smart said. "But if an attacker has compromised your pipeline before that stage, then congratulations -- you've just programmatically signed my malware, and now everyone is going to trust it more."
Lessons learned
Gazdag emphasized that none of the CI/CD pipeline engagements NCC Group worked on involved any exotic exploits or novel techniques. "None of these problems are new, and we didn't use any zero days," he told the session audience.
In an interview with SearchSecurity, Gazdag said there were a few customer engagements where the research team couldn't break into the pipelines. "When we couldn't compromise a pipeline, it's because the company had the security controls that we either recommend or were previously deployed," he said.
Those controls include relatively simple steps like segmenting of production environments, applying principle of least privilege, using role-based access control, and setting up alerts and proper logging for the pipelines. Gazdag and Smart also encouraged organizations to limit pipeline access to external dependencies that have been validated.
Smart told SearchSecurity that many of the customers NCC Group worked with didn't fully grasp how big of an attack surface their CI/CD pipelines were -- and how much damage could be done if they were compromised.
"These pipelines are, by their very nature, privileged. The more automation you add in, the more privileges you need to give your automation," he said. "If you've got a box that can automatically do things for you, and someone compromises the box, then that attacker has all of the permissions that the box had."
