Codecov breach raises concerns about software supply chain

A Codecov breach disclosed earlier this month may have far-reaching implications on the software supply chain.

On April 15, the code coverage vendor disclosed that its Bash Uploader script, which is used for sending coverage reports, had been breached by a threat actor, who then modified the script to gain access to and export data from users' continuous integration (CI) environments. The initial breach occurred on Jan. 31, according to the disclosure, and Codecov became aware of the breach on April 1.

"Immediately upon becoming aware of the issue, Codecov secured and remediated the affected script and began investigating any potential impact on users. A third-party forensic firm has been engaged to assist us in this analysis. We have reported this matter to law enforcement and are fully cooperating with their investigation," the disclosure read.

While Codecov took quick action following its discovery, the potential scope of the attack quickly grew. According to a Reuters report on April 20, anonymous sources investigating the incident said the breach resulted in threat actors gaining access to hundreds of the vendor's customers. Moreover, some of those customers are large vendors themselves -- extending the potential supply chain threat even further.

Cloud infrastructure vendor HashiCorp was the first Codecov customer to come forward with a breach disclosure on April 22, when director of product security Jamie Finnigan posted on HashiCorp's discussion forums that the company "was impacted by a security incident with a third party (Codecov) that led to potential disclosure of sensitive information." Finnigan confirmed the connection to the Codecov breach disclosed on April 15 and said that HashiCorp rotated one of its GPG private keys used for release signing and verification.

But nearly two weeks after the Codecov breach came to light, HashiCorp remains the only customer to disclose a breach connected to Bash Uploader attack. Codecov declined SearchSecurity's requests for comment on HashiCorp's announcement. Instead, a spokesperson responded with the company's original April 15 disclosure statement.

UPDATE: Codecov published an update to its security advisory Thursday with indicators of compromise (IOCs) for the breach as well as additional information about impacted customers. The IOC section includes the modified portion of the Bash Uploader script that led to compromises of Codecov customers and IP addresses associated with the modified script. The update said Codecov is confident that there was only only one change made to Bash Uploader, and that the program was affected during "108 windows of time" between Jan. 31 and April 1.

"We have recently obtained a non-exhaustive, redacted set of environment variables that we have evidence were compromised. We also have evidence on how these compromised variables may have been used. Please log-in to Codecov as soon as possible to see if you are in this affected population," the update reads.

The update also said Codecov has notified users believed to be affected by the breach and has posted a notification in its app. The company offered mitigation steps for customers who used Bash Uploader between  Jan. 31 and April 1 and did not conduct a checksum validation of the program. Those steps include re-rolling all credentials, tokens, or keys located in the environment variables of their CI process.

Security research Ax Sharma noted in a blog post Tuesday that both Netflix and operators of the text editor Vim also rotated their credentials as a precaution. It's unclear if either organization has found evidence of intrusion; neither responded to SearchSecurity's request for comment.

The Codecov breach comes five months after another supply chain attack disclosure: SolarWinds.

In December, Russian nation-state threat actors compromised software updates for SolarWinds' Orion IT management platform. Many organizations were compromised as a result, including a number of agencies in the United States federal government. The Codecov attack's scope and impact are still unknown at this stage, but some experts have expressed grave concern.

Volexity director of research Andrew Case tweeted Tuesday that compared to the SolarWinds hack, which carried one potential source of security issues, Codecov is "a recursive nightmare of internal dev systems, CI systems, production systems then loop for all deployed 3rd party software."

Anonymous security researcher x0rz also had strong words for the attack's scope.

"Codecov is the final boss of supply chain: it gives access to a variety of other dev projects, it's an endless graph of interdependent code. At this point the real question is, who isn't impacted by this?" X0rz tweeted.

Sophos principal research scientist Chester Wisniewski told SearchSecurity that the fallout of the attack, compared to SolarWinds, will depend on how Codecov customers respond.

"It isn't likely to have impacted as many organizations [as SolarWinds]; yet, because of the nature of the Codecov hack, we may never really know. I certainly consider it to be a similar level of seriousness and concern," he said. "It will all come down to how Codecov customers respond, rotate credentials out and carefully investigate actions that may have been undertaken during the window we believe the attacks were carried out."

Wisniewski also said that determining scope may prove to be a challenge.

"This type of supply chain attack is not as valuable to the average criminal as it would be to a nation-state. This could also be much harder to gauge the scope of the attack because there aren't artifacts of a malware infection, like SolarWinds. The attackers may have acquired access keys, passwords and API tokens that could be used to further compromise Codecov customers' networks and tools that may then be used in further supply chain poisoning," he said. "We may never fully understand the scope of this hack."

Alexander Culafi is a writer, journalist and podcaster based in Boston.