Getty Images/iStockphoto
Azure Pipelines vulnerability spotlights supply chain threats
Legit Security researchers discovered a remote code execution flaw within Microsoft's Azure DevOps platform that could give threat actors complete control of development pipelines.
Researchers with Israeli startup Legit Security discovered a vulnerability in Microsoft Azure Pipelines that could let threat actors submit malicious code to development workflows and launch supply chain attacks.
In a blog post on Thursday, Legit Security revealed the technical details of CVE-2023-21553, a high-severity vulnerability affecting Azure DevOps Server that Microsoft patched in February's Patch Tuesday release. The vulnerability, which allows remote code execution (RCE), lets attackers gain complete control of variables and tasks within Azure Pipelines, which automatically builds and tests production code within the Azure DevOps platform.
The Azure Pipelines flaw affected both the SaaS version of Azure DevOps Server and the self-hosted, on-premises version. While the cloud version was instantly fixed and required no customer actions, customers running the on-premises version need to patch their instances to remediate the RCE vulnerability.
Failure to patch could result in serious consequences, according to Legit Security's report. In the blog post, Nadav Noy, a Legit Security researcher, explained how exploiting the vulnerability gives attackers a host of options, from obtaining code to engineering extensive supply chain attacks.
"An attacker could use commit messages or any other user-controlled variable to inject logging commands that will result in overwriting existing variables and pipeline takeover," Noy wrote. "The attacker could escalate this vulnerability to gain lateral movement by leveraging exfiltrated secrets and access keys."
Exploiting the vulnerability
Liav Caspi, co-founder and CTO of Legit Security, told TechTarget Editorial the discovery of the Azure Pipelines flaw stemmed from company's research team exploring ways to submit code into production pipelines on various platforms, including Azure DevOps.
"The question was, 'If someone can hack into your pipeline, what's the biggest damage that they can cause?'" Caspi said.
Legit Security researchers discovered a flaw in the variables mechanism within Azure DevOps that let them abuse logging commands and gain control of pipelines. "If a malicious actor can get their carefully crafted string printed or echoed, they can gain access to variables and tasks in the pipeline," Noy wrote.
An attacker would first need to find pre-defined variables that can be externally controlled. Legit Security researchers looked through Microsoft's list of pre-defined variables and found that build variables fit the bill. "These are the only variables that can be influenced by an actor with no direct access to the Azure Pipelines environment," he wrote. "An attacker only needs permission to create a pull request or push a commit to exploit this vulnerability."
The researchers then altered a build variable by injecting a logging commands syntax. From there, they made a commit message that altered certain variables, letting them change a download URL and instead link to a malicious bash script, which compromises the pipeline.
Noy explained the research team found existing software projects that might be vulnerable to the attack. One example was Scikit-learn, an open source machine learning framework for Python that has more than 24,000 forks and 50,000 stars on GitHub.
Legit Security researchers found they could attack the Scikit-learn pipeline by forking the repository, creating an innocent-looking pull request and then injecting malicious content in the last commit message. If a maintainer review doesn't spot the suspicious pull request, then it will be merged with the main branch, allowing threat actors to executing malicious logging commands.
"Once an attacker finds a vulnerable pipeline, like in this example, they could read all secrets available to the pipeline job, which are usually very sensitive and contain API keys, database passwords, or cloud credentials," Noy wrote. "Another option is to modify the build output and carry on a supply chain attack -- just like in the SolarWinds incident."
Legit Security initially reported CVE-2023-21553 to Microsoft on Sept. 22. The software giant exchanged information with researchers a few days later and awarded a bug bounty reward to Legit Security. Caspi said the two companies worked closely over several months to not only fully address the vulnerability but also explore the underlying issues.
While a patch was released last month for Azure DevOps Server, Legit Security agreed to Microsoft's request to withhold the technical details of the attack technique until Thursday so that organizations running on-premises Azure Pipelines could have at least 30 days to update their software.
Highlighting supply chain threats
While RCE vulnerabilities typically receive critical severity scores between 9 and 10 in the CVSS, the Azure Pipelines flaw received a 7.5 score, which is high severity. While Legit Security described the vulnerability as "widely exploitable," Caspi said exploitation isn't a simple process and depends on several factors.
"This is a very elaborate and very sophisticated attack," he said, "but it can happen."
Microsoft agreed in its security advisory for CVE-2023-21553, labeling the attack complexity "high." While no exploitation has been detected in the wild, Caspi said the vulnerability highlights weaknesses in development environments that, if unresolved, could be exploited by threat actors in a campaign similar to the supply chain attack on SolarWinds. In that incident, Russian nation-state threat actors compromised SolarWinds' development environment and injected a backdoor into software updates for Orion IT management software platform.
Caspi said traditionally, there wasn't a lot of security focus on build systems prior to the SolarWinds attacks; that's still mostly true today. "I would say there's a lot of negligence around development infrastructure," he said. "Since SolarWinds, there have been growing calls for adding zero trust to the build systems so you can secure it like the production environment. But it still doesn't get a lot of focus from the community."
Even if zero trust was applied in this case, Caspi said improvements wouldn't necessarily have prevented exploitation of the Azure Pipelines flaw because no user authentication is required. The problem, he said, is that many automated pipelines themselves are designed to implicitly trust inputs.
IDC analyst Katie Norton said vulnerability highlights a lack of human oversight in most continuous integration (CI) and continuous delivery (CD) pipelines.
"To me, what is most significant is that these types of pipeline exploits is that risk is compounded by the fact that CI/CD systems are highly automated and often have little human involvement. So malicious code or artifacts could push their way through to production without any review or approval," she said. "I would say this is significant enough that organizations with Azure DevOps Server will want to prioritize this patch or, at a minimum, make sure they have some compensating controls in place."
However, Norton also noted that responding to vulnerability disclosures is a significant problem for developers. According to a recent IDC survey, respondents said the biggest challenge they face in terms of DevOps tooling gaps and exposure was the inability to quickly patch critical vulnerabilities.
Melinda Marks, senior analyst at TechTarget's Enterprise Strategy Group, said the vulnerability underscores the importance of securing pipelines and the challenge of fully patching on-premises development infrastructure because of the complexity of software supply chains.
"It's hard to find and remediate this type of vulnerability or an attack that uses the features of the pipeline to get in and wreak havoc," Marks said. "As the Legit blog points out, printing and logging untrusted data is not considered a security risk."
Fernando Montenegro, an analyst at Omdia, said the Azure Pipelines vulnerability demonstrates that there's more to securing the software supply chain than just validating external components. "Flaws within that pipeline, such as the one demonstrated here, should be treated as high-priority incidents since they have the potential for significant compromise down the line."
On the positive side, Montenegro commended Microsoft and Legit Security for working together to address a complex vulnerability. "We're now at a time in industry where this kind of collaboration can yield fixes that are helpful for users," he said.
