peshkova - Fotolia
Operation Rosehub patches Java vulnerabilities in open source projects
Google employees recently completed Operation Rosehub, a grass roots effort that patches a set of serious Java vulnerabilities in thousands of open source projects.
In response to a nearly ubiquitous set of Java vulnerabilities, Google employees recently completed Operation Rosehub, a 12-month project to patch the thousands of open source projects exposed through their use of widely-used collections of reusable Java code.
The Java vulnerabilities were dubbed "Mad Gadget" because they existed in seven Java "gadget" classes in three versions of the popular Apache Commons collections. These vulnerabilities first surfaced in November of 2015, and were initially thought to affect thousands of Java apps, including Oracle WebLogic, IBM WebSphere, JBoss, Jenkins and OpenNMS.
"Mad Gadget is one of the most pernicious vulnerabilities we've seen," because it makes "object deserialization for the entire JVM process Turing complete with an exec function," wrote Justine Tunney, software engineer on Google's TensorFlow project, in a blog post on Operation Rosehub. Tunney noted that Oracle, Cisco, Red Hat, Jenkins, VMware, IBM, Intel, Adobe, HP and SolarWinds had all been affected by the Java deserialization vulnerabilities.
"Since many business applications use object deserialization to send messages across the network, it would be like hiring a bank teller who was trained to hand over all the money in the vault if asked to do so politely, and then entrusting that teller with the key," Tunney wrote. "The only thing that would keep a bank safe in such a circumstance is that most people wouldn't consider asking such a question."
Open source Java vulnerabilities found
Five months after the Java vulnerabilities were first made public, Tunney became aware of high-profile open source libraries that had not yet been patched, and she began sending pull requests to unpatched projects on GitHub. The patches in most cases required just a single line change, which was relatively easy to do with GitHub's GUI.
According to Tunney, as more Google employees were recruited to the effort, it became "apparent that the problem was bigger than we had initially realized." While patching framework projects used by other applications, they realized that they would need to patch those other dependent projects as well.
"This was when we realized the particularly viral nature of Mad Gadget," Tunney wrote. "We came to the conclusion that, in order to improve the health of the global software ecosystem, the old version of Collections should be removed from as many codebases as possible."
Ultimately, the group that became Operation Rosehub searched all public open source code on GitHub and uncovered "2,600 unique open source projects that still directly referenced insecure versions of [the Apache Commons] Collections," Tunney wrote.
"Ultimately, security rests within the hands of each developer. However we felt that the severity of the vulnerability and its presence in thousands of open source projects were extenuating circumstances. We recognized that the industry best practices had failed. Action was needed to keep the open source community safe. So rather than simply posting a security advisory asking everyone to address the vulnerability, we formed a task force to update their code for them. That initiative was called Operation Rosehub."
The operation could be considered an important success for open source software. "We want to draw attention to the fact that the tools now exist for fixing software on a massive scale, and that it works best when that software is open," Tunney wrote.
"The success of Google's Rosehub effort is extraordinary, and must become ordinary," Paul Vixie, CEO of Farsight Security, told SearchSecurity. "Every dev team in the world depends on the security of the rest of the infrastructure, and Google's demonstration shows the kind of barn raising community spirit which built the internet in the first place. We can either fix these things upfront, or we can live with the consequences, including the fact that fixing code after it has shipped is much more difficult."
However, because most of the patches required a simple change to a single line of code, it is not clear that the open source approach would succeed for more complicated vulnerabilities.
"This solution is certainly interesting and can be used to fix a very specific type of software bug where simple string matches and replaces can find vulnerable code and replace it easily," John Bambenek, threat systems manager at Fidelis Cybersecurity based in Bethesda, Md., told SearchSecurity. "Other issues are probably too complex for this kind of solution, but the large scale scanning of code for issues is certainly an approach that can be used to great effect to fix the low hanging fruit."
While Operation Rosehub appears to have remediated the Mad Gadget Java vulnerabilities in thousands of Open Source projects, there is still an enormous challenge to mitigate these Java vulnerabilities in software that has already been deployed on systems and devices that are not -- and perhaps cannot -- be patched.
"Unfortunately, hundreds of millions of devices include the Mad Gadget vulnerability and will never be patched -- we're going to have to wait for them to wear out and be removed due to natural causes," Vixie said. "While the law requires that consumers be told the ingredients in our food and the active ingredients in our medicines, no law requires that buyers be informed of the digital ingredients of devices and programs. This means we will learn of unpatched devices one attack at a time."
