Aqua Security warns of significant risks in Prometheus stack
The cloud security vendor called on Prometheus to provide users with additional safeguards to protect against misconfigurations discovered in the open source monitoring tool.
Aqua Security warned that at least 336,000 exposed Prometheus instances are vulnerable to alarming security flaws that could allow attackers to steal credentials and API keys, as well as launch DOS attacks.
Assaf Morag, director of threat intelligence at Aqua Security, and Yakir Kadkoda, lead security researcher at Aqua, published new research Thursday detailing several vulnerabilities and security flaws found in Prometheus, a popular open source monitoring and alert tool. The research highlighted three significant security risks, including information disclosure, DOS and remote code execution, that are associated with publicly accessible Prometheus servers and exporters.
Despite prior warnings and increased awareness around the risks of unauthenticated Prometheus servers, they continue to be present in user environments. Aqua researchers conducted a Shodan search and found at least 336,000 internet-exposed Prometheus exporters and servers, with the U.S. claiming the top spot among several countries.
"When Prometheus servers or exporters are connected to the public internet without authentication, they introduce a significant risk. Such misconfigurations allow anyone to query the exposed environments to list labels or metrics. Attackers can exploit this access to gather seemingly trivial data and, with the help of secret-scanning tools, uncover sensitive information, including credentials, passwords, authentication tokens, and API keys," Morag and Kadkoda wrote in the blog post.
The researchers discovered several examples of information disclosure risks related to unauthenticated Prometheus servers and warned it is "not a theoretical risk." The blog post highlighted how exposed Prometheus servers and the metrics can reveal sensitive company data such as subdomains, Docker registries and images.
Aqua provided one real-world example and said the company addressed the risk by blocking the vulnerable instance.
"For instance, we discovered an unauthenticated Prometheus instance associated with Skoda, one of the biggest European car manufacturers. In addition to exposing Docker registries and images linked to Skoda, this Prometheus server also revealed subdomains and paths of Skoda through the kube_ingress_path metrics," the blog post said.
Aqua also detailed the risks of DOS attacks. Researchers noted the Go debugging interface, known as pprof package, is commonly used for performance profiling. "Misconfigured Prometheus servers and exporters exposed to the internet provide HTTP access to the pprof endpoint, which is enabled by default in most Prometheus components," the blog post said.
While the /debug/pprof endpoint is designed to help users with remote profiling, the researchers found that attackers could exploit it to conduct DoS attacks, and victim organizations could incur service outages. The risk increases if proper access controls or resource limits are not implemented on Prometheus instances, they warned.
The blog post highlighted prior warnings of /debug/pprof exposures. After conducting an audit in 2020, Cure53, a Germany-based cybersecurity vendor, previously reported the flaw to Prometheus and flagged it again in 2022 on GitHub. However, the issue wasn't addressed.
"Exposing Prometheus, its components, and various exporters to the internet without authentication is considered poor practice. However, the /debug/pprof endpoint introduces a particularly concerning risk: the ability to directly impact the host machine/pod and serve as a vector for DoS attacks. In our view, this vulnerability demands attention and mitigation," the blog post said.
Aqua disclosed its findings to Prometheus and said it received the following response: "Supporting good production practices trumps protecting users from gross misconfigurations."
Repojacking risks
Regarding code execution, the researchers found risks related to "repojacking" attacks. As an open source monitoring tool, Prometheus uses GitHub as its code repository. GitHub has proven to be a popular target for attackers in recent years for supply chain attacks.
"GitHub RepoJacking is a type of supply chain attack that allows attackers to takeover GitHub projects' dependencies or an entire project to run malicious code on whoever uses these projects. It occurs when an owner deletes or renames a repository," the blog post said. "If this repository is still referenced by other internal or external projects, an attacker can create a new repository with the same old name (changed or deleted) and by doing so take control over the referenced name."
The researchers examined Prometheus's official documentation and discovered several exporters, which collect metrics from the monitored endpoints, were vulnerable to repojacking attacks. Attackers could manipulate the exporter name to host a malicious version and then trick users into cloning and deploying it. The researchers warned that could lead to remote code execution on the victim organization's system.
Kadkoda told Informa TechTarget Editorial that repojacking poses challenges for many open source projects. While awareness is increasing around the issue and GitHub implemented measures to mitigate the risk, he said the threat still persists. "Addressing this issue requires greater responsibility from maintainers. They should regularly review and verify their references and links to ensure they do not point to repositories that could be compromised," Kadkoda said.
Regarding the overall authentication challenges, Kadkoda said although Prometheus and its components support TLS encryption, including client certificate and basic authentication, the features are not enabled by default. "Consequently, the lack of default authentication and improper configurations often result in users overlooking these critical security measures," he said.
To combat the risks, Aqua recommended implementing authentication to ensure that only authorized users can access sensitive data, limiting external exposure and monitoring debugging environments.
While Prometheus appeared to dismiss Aqua's concerns, Kadkoda pushed back on the organization's response. He acknowledged that Prometheus is an open source project and applauded the community's maintenance effort but said users require more help to address security risks.
"While we understand the importance of encouraging best practices, it's equally important to consider the reality that users may make the same mistakes repeatedly. Overlooking such scenarios, especially when they can lead to significant security vulnerabilities, may leave users unnecessarily exposed," Kadkoda said. "We believe that balancing good production practices with safeguards against common misconfigurations is essential for fostering a secure and reliable ecosystem. This is particularly true for open source projects that are widely adopted by users with varying levels of expertise."
Arielle Waldman is a news writer for TechTarget Editorial covering enterprise security.
