JetBrains, Rapid7 clash over vulnerability disclosure policies
In a blog post this week, JetBrains argued that attacks on TeamCity customers were the result of Rapid7 publishing the full technical details of two critical vulnerabilities.
A dispute between software maker JetBrains and security vendor Rapid7 has highlighted ongoing concerns with coordinated vulnerability disclosure policies and practices.
On March 4, JetBrains disclosed two critical vulnerabilities tracked as CVE-2024-27199 and CVE-2024-27198 that allow for authentication bypass against on-premises TeamCity servers. The following day, JetBrains and Rapid7, credited for discovering and reporting the flaws, confirmed that exploitation activity had begun against vulnerable servers. However, a disagreement over the disclosure process came to light.
In a blog post on March 4, Rapid7 accused JetBrains of breaking the coordinated vulnerability disclosure process and attempting to silently patch the vulnerabilities with the release of TeamCity 2023.11.4. Rapid7 included the full technical details of the vulnerabilities in its blog post and also published proof-of-concept (PoC) exploits.
Rapid7 also explained in the post that in disclosure communications with JetBrains during February, the company had proposed releasing the patches for CVE-2024-27199 and CVE-2024-27198 privately before publicly disclosing the flaws. Rapid7 rejected the proposal, emphasizing its policy against silent patching because it believes doing so would put customers at risk.
Daniel Gallo, TeamCity solutions engineer, addressed the dispute in a follow-up post on March 5 in which he admitted that JetBrains broke off communication with Rapid7 following the rejection of that proposal.
"At this point, we made a decision not to make a coordinated disclosure with Rapid7 as we strongly believe that publishing all technical details at the same time as releasing a fix allows anyone to immediately exploit the issue before all customers have had a chance to patch their servers," Gallo wrote.
Further complicating matters is the fact that the two companies can't seem to agree on when TeamCity 2023.11.4 was officially released. JetBrains, headquartered in Prague, told TechTarget Editorial it was released on March 4 at 3 p.m. Central European Time. Rapid7, however, said it was released on March 3.
The dispute reached a boiling point this week when Gallo published another blog post on Monday, titled "Preventing Exploits: JetBrains' Ethical Approach to Vulnerability Disclosure." The post put further blame on Rapid7 for attacks on TeamCity customers, which Gallo said began after the cybersecurity vendor's full disclosure. "This was due to the immediate availability of publicly documented exploit examples published by Rapid7, which meant attackers of any skill level had all the resources they needed to quickly exploit the vulnerabilities in the wild," he wrote.
Gallo added that JetBrains believes the simultaneous release of patches with full technical vulnerability details "can lead to more harm than good." Like others in the industry, he argued that full disclosure could give attackers a heads-up on how to exploit the flaws.
"We are aware of many customers who were able to apply the security patch or upgrade prior to the exploits being published by Rapid7," Gallo wrote. "Unfortunately, many others were not as fortunate."
Gallo added reported attacks from four unnamed customers, two of which involved ransomware. "Files on their TeamCity server were all encrypted and a ransomware note was left on the machine," the blog said in reference to "Customer A."
In a blog post on March 8, GuidePoint Security researcher Drew Schmitt revealed that BianLian exploited the TeamCity vulnerabilities to gain initial access to a victim organization's environment. Once successful, BianLian operators deployed a PowerShell Go backdoor, GuidePoint observed.
"As we have seen throughout 2023 and into 2024, BianLian continues to prove how they can adapt to a changing environment, especially in regards to the exploitation of emerging vulnerabilities," Schmitt wrote in the blog.
Silent patching debate
Silent patching concerns have been raised by many infosec professionals and vendors like Rapid7 throughout the years. For example, in 2022, Tenable CEO Amit Yoran accused Microsoft of silently patching on many occasions. Tenable researchers had recently reported Azure flaws and expressed frustration with the disclosure process, which Yoran said lacked transparency.
The ongoing feud between JetBrains and Rapid7 shows that researchers and vendors remain divided on how best to disclose vulnerabilities without giving attackers an advantage.
Bob Huber, chief security officer and head of research at Tenable, told TechTarget Editorial that he believes JetBrains was naive to think the flaws were unknown prior to disclosure, or that no actor had been exploiting them previously. He added that JetBrains software is a popular target for attackers.
For example, in December, CISA issued a joint advisory warning that a Russian nation-state threat actor, commonly known as APT29 or Cozy Bear, had exploited a different TeamCity vulnerability against several customers, including technology companies. Months before that, Microsoft confirmed that a North Korean nation-state actor exploited a TeamCity remote code execution vulnerability, tracked as CVE-2023-42793, that also allowed for authentication bypass.
Huber said one hazard of vendors' silent patching practices is security leaders being left in the dark regarding exposure to risk. That lack of threat intelligence can lead to breaches and data theft, he warned. Huber added that providing full transparency enables organizations to investigate and resolve issues before attackers have a chance to act.
"By sharing limited details on the vulnerabilities and dismissing coordination efforts with the researchers, JetBrains created more work for its customers, sending them off on a wild goose chase trying to understand where they are vulnerable," Huber said. "This isn't JetBrains' first vulnerability disclosure exercise. Security researchers and adversaries will reverse-engineer the vulnerability anyway, so their actions only delay the inevitable."
Dustin Childs, head of threat awareness for Trend Micro's Zero Day Initiative (ZDI) research team, said researchers and vendors have long disagreed over the best way to disclose vulnerabilities. He stressed that ZDI has had its fair share of disagreements with vendors over disclosures, including one with Microsoft last year. ZDI's default policy provides vendors with 120 days before public disclosure, but Childs said it does not have a strict policy on publishing details after patches become available.
Regarding the JetBrains and Rapid7 case, Childs highlighted Gallo's March 5 blog post in which he said JetBrains opted not to make a coordinated disclosure with Rapid7. Childs said that statement negates JetBrains' claims about it practicing ethical vulnerability disclosure.
"You can't practice coordinated disclosure only when it benefits you. There's a difference between coordinated disclosure and controlled disclosure, and it seems they were looking to control the narrative rather than coordinate with Rapid7," Childs told TechTarget Editorial. "They also seem to underestimate how quickly reverse-engineers can patch diff and create exploits. In my experience, the fastest I've seen was a scant four hours."
Responsible vs. coordinated disclosure
In addition to blaming Rapid7 for TeamCity attacks, Gallo also argued that JetBrains' disclosure policy is "commonplace" in the industry and referred to Microsoft's and Google's own policies. While it appeared that those remarks were meant to strengthen JetBrains' argument, Childs said referencing those companies shows JetBrains might be focused more on brand reputation than customer protection.
"Microsoft and Google have disclosure policies that include their best interests," he said. "They often want to protect the brand as much as the customer. Research organizations like Rapid7 have different priorities and thus different disclosure policies."
Childs attributed perpetual discussions on the subject to an industrywide shift away from the term responsible disclosure to coordinated disclosure. He acknowledged that some infosec professionals might have viewed Rapid7's release of full technical details as "irresponsible." However, Childs emphasized that threat actors don't wait for organizations' patch management issues to be resolved; timely patching remains an ongoing struggle for enterprises of all sizes.
He also said JetBrains' decision to patch silently and leave Rapid7 out of the disclosure process would be considered unethical by many infosec professionals.
"What we need to remember is that threat actors share information when it benefits their needs," Childs said. "Withholding information and pointing fingers doesn't make anyone safer. It's not the researchers or vendors that suffer the consequences -- it's the end users."
Jake Williams, a faculty member at IANS Research, also criticized JetBrains for blaming Rapid7 for the attacks against its customers. He interpreted the finger-pointing as JetBrains distracting from two real issues. First, like Huber, he emphasized that TeamCity is a known target of threat actors. Second, Williams said JetBrains might have been deflecting the fact that its servers contained trivially exploitable authentication bypass vulnerabilities.
"I have no doubt that TeamCity customers have been impacted negatively by Rapid7 following its published disclosure policy. But none of this happens if JetBrains' code wasn't vulnerable in the first place," Williams told TechTarget Editorial. "I'd like to see JetBrains redirect the energy they put into the blog post calling out Rapid7 toward engineering a vulnerability-free product."
Nate Warfield, director of threat research and intelligence at security platform provider Eclypsium, said the definitions of ethical and responsible disclosure can differ depending on the vendor and researcher. "Using ethical to describe a vulnerability disclosure process is merely an evolution of the much-despised Microsoft term responsible disclosure," Warfield told TechTarget Editorial.
He acknowledged that many vendors and professionals support releasing a full PoC exploit with the mindset that it will help organizations determine if they're affected by the vulnerability and develop defensive strategies. In addition, Warfield highlighted how PoCs benefit the greater security community by providing technical details that aid in threat intelligence gathering.
However, he said PoCs lack detailed security recommendations and can leave organizations in a race against time with the attackers. Similarly to Huber and Childs, Warfield emphasized that exploitation campaigns can start within hours, and depending on the time zone, organizations might not see advisories until after their devices have been affected.
Because of the short window, he believes it's unfair to blame organizations for a lack of timely patching.
"The problem is, rarely do we see research companies release said defensive guidance -- their blogs go deeply technical covering every aspect of what caused the vulnerability, the offending code, their research process and the exploit. However, something as simple as the log messages thrown by the system when it's exploited or a rough detection rule for any of the most common security tools is rarely, if ever, included," Warfield said.
Regarding the disclosure argument between JetBrains and Rapid7, Warfield believes the software developer appeared to take the reported vulnerabilities seriously and understood how the PoC fallout posed a potential risk to customers. In addition, he said the silent patching accusations against JetBrains could be fruitless.
"This generally refers to shipping an update without assigning a CVE, yet JetBrains had already assigned two CVEs to the report days after receiving it," Warfield said. "The inference here is they asked Rapid7 to withgo releasing the full write-up and exploit until their customers had a few days to patch. This is not an unreasonable request. JetBrains even went so far as to try and obfuscate the patch to make the inevitable patch diffing and reverse-engineering more complex, buying their customers additional hours or possibly days to remediate."
On the other hand, he said it's unclear whether that approach was communicated with Rapid7. Though the silent patching accusations are murky, Warfield said it's obvious that JetBrains did not follow a coordinated disclosure process, which prompted Rapid7 to disclose the full details.
Warfield emphasized how important it is for security researchers to work closely with the vendors whose products they break. Researchers save vendors time, he said, by disclosing exploits and advanced tactics attackers could leverage against customers. He also recognizes how essential it is to allow vendors time to protect their customers. Overall, both sides should strive to do better, Warfield stressed.
"Like it or not, the work done by the security research community is weaponized against organizations on a daily basis," he said. "In this specific instance, it was only a few weeks from report to patch availability, and with better communication or taking a real-world approach to the disclosure, it could have resulted in fewer organizations being impacted."
Arielle Waldman is a news writer for TechTarget Editorial covering enterprise security.
