SaaS CI/CD vendor CircleCI urged customers to rotate all secrets data, the latest of several security breaches weighing on SecOps pros charged with responding.
Editor's note: This story was updated to reflect additional information issued by CircleCI on its official blog post on Jan. 7 and Jan. 12, as well as an incident report released by the vendor Jan. 13.
SecOps pros in CircleCI environments potentially faced hours of work to rotate all secrets data stored in their repositories in response to a security advisory from the SaaS CI/CD vendor first issued Jan. 4.
CircleCI issued its first statement on its official blog and via email to users about a security incident it was investigating between Dec. 21 and Jan. 4. The statement recommended that users "immediately rotate any and all secrets stored in CircleCI."
The recommendation was made out of "an abundance of caution," according to the statement, but no further details about the scope of the incident or how CircleCI has addressed it internally were given Jan. 4. The vendor followed this with a set of instructions Jan. 5 for how to perform a secrets rotation.
Around 7 p.m. ET Jan. 5, CircleCI provided an update on its official blog post that included a link to an open source tool for discovering secrets in CircleCI environments in response to customer requests and also made audit log access free for all customers up to 30 days.
SecOps pros were concerned about the lack of detail in CircleCI's communications so thus far about a recent security incident.
Another post made Jan. 7 also said CircleCI had rotated all the access keys in its production environment and completed an audit of all system access, and re-emphasized that customers should "rotate any and all secrets stored in CircleCI" for all projects. These include OAuth tokens; project and user API tokens; project environment and context variables; project SSH keys; and runner tokens, according to a detailed list included in the update.
On Jan. 13, CircleCI issued a detailed post-mortem report about the incident. The report said the company's investigation into the breach began when a customer reported suspicious activity involving a GitHub OAuth account Dec 29. CircleCI's investigation, which took until Jan. 4 to determine whether other customers had been affected, revealed that one of its developers' privileged accounts had been accessed Dec. 16 by an unauthorized third party.
"An unauthorized third party leveraged malware deployed to a CircleCI engineer's laptop in order to steal a valid, [two-factor authentication(2FA)]-backed [single sign-on] session," the post-mortem report stated. "The malware was able to execute session cookie theft, enabling [the attacker] to impersonate the targeted employee in a remote location and then escalate access to a subset of our production systems."
The systems to which the attacker gained access included databases and data stores that contained customer data, including environment variables, tokens and keys, generally referred to as secrets. The attacker also accessed encryption keys that allowed them to steal even encrypted secrets, according to the postmortem. On Jan. 4, CircleCI directed customers to rotate, or change, all such data they had stored within their CircleCI CI/CD SaaS account. The last record of unauthorized access to CircleCI's systems uncovered by its investigation took place Dec. 22.
CircleCI drew criticism for lack of detail in initial reports
In its Jan. 7 update, CircleCI also addressed criticism for issuing its Jan. 4 advisory well after U.S. East Coast business hours. "We understand that many of our North American customers experienced late nights and on-call rotations once our guidance to rotate secrets was released at 6:30 p.m. PT / 9:30 p.m. ET on Wednesday, January 4," the update stated. "We erred on the side of getting information out as fast as possible to minimize any potential exposure time."
When the initial recommendation was issued Jan. 4,many SecOps teams began the work of sifting through software development project repositories to find and update secrets, or privileged credentials used to authenticate and authorize access to systems.
One CircleCI user in the U.K. posted on social media that he'd been up doing this work at 4 a.m. Another responded to CircleCI's Twitter post,calling for the company to make this information easier to locate in its UI.
"It's certainly disruptive to end users and admins alike," said Peter Wright, a systems engineer for a CircleCI customer in Los Angeles. "It also will trigger lots of work investigating if any potential break-ins happened during the window they stated … so it has a domino effect on support, admins, developers and security people at affected companies."
In CircleCI's platform, secrets can take the form of personal or project credentials, including API tokens used to broker access to specific software development projects. Project API tokens across the board were "invalidated" and will need to be replaced, according to the initial CircleCI advisory.
CI is not only a critical component of many environments for productivity purposes, but it also lives in a sensitive part of your infrastructure.
Peter WrightSystems engineer, CircleCI user
The potential effect of the incident on API access to source code is of particular concern, Wright said.
"CI is not only a critical component of many environments for productivity purposes, but it also lives in a sensitive part of your infrastructure," he said. "It often has access both to your source code, as well as having the ability to deploy software. So it's not unreasonable to worry that any stolen secrets could be used to gain access to customer environments in a privileged way."
In a separate blog update Jan. 12, CircleCI confirmed that some customers' AWS security tokens had been compromised as a result of the breach. AWS began notifying these customers via email that day.
"At this time, there is no indication that your AWS account was accessed," CircleCI's Jan. 12 update stated. "Only that there is a possibility the token stored in CircleCI was leaked, and therefore should be deleted from AWS and rotated."
CircleCI post-mortem reflects fallibility of passwords: analyst
This is not the first CircleCI security incident to be reported in recent years. In 2019, it disclosed an incident involving a third-party analytics vendor in 2019. That disclosure contained specific details about the scope of the systems affected and how CircleCI had responded internally, information that was missing 24 hours after the Jan. 4 advisory.
"What makes this so scary is … not a lot of details from CircleCI yet on the scope of what happened, what steps they have done to remediate it and what signatures we can look for, [such as] IP addresses or things like that that an attacker may be using stolen credentials with," Wright said on Jan. 5.
This information will be crucial for SecOps pros to prioritize their response to this incident, said Melinda Marks, an analyst at Enterprise Strategy Group, a division of TechTarget.
"Just like if there is a breach of a company, if you use their service, you would go change your password," Marks said. "In the case of the CircleCI breach, you don't know how much access the attacker has to the repos, so you'd want to rotate all your secrets to minimize risk of the attacker … gaining access to your repo and code."
Automatically rotating secrets, especially in cloud-hosted services, is a best practice for security hygiene, but like the timely application of software patches, isn't always followed within enterprise organizations. Storing secrets in code repositories is also best avoided, but while 83% of 350 respondents to a recent ESG survey of enterprise organizations scan their Git repositories for risky secrets in code, 31% have had security incidents resulting from exposed secrets.
"People do typically scan, but they don't know how to prioritize taking action, [and] a high percentage are still getting secrets stolen," Marks said.
Following the Jan 13 post-mortem, another analyst said that this breach shows it's time for the tech industry to move beyond passwords, even those protected by 2FA.
"The biggest takeaway from the incident is that compromised developer credentials are a significant attack vendor," said Katie Norton, an analyst at IDC. "Logging in is way easier, and often more likely to go undetected, than hacking in, and this example is a good lesson that 2FA is not a panacea."
"Look at LastPass, they had a huge breach, and they themselves are a company whose aim is to make passwords more secure for consumers and enterprises," Norton said. "I am starting to hear more on the front of passwordless authentication and other methods for credentials that are trying to solve this problem, but I think technologies like that will take a while to gain critical mass, since we are so password-driven currently."
In the meantime, users can ensure the passwords and other credentials they use are ephemeral, or short-lived, to mitigate similar future attacks, Norton said. Ephemeral identity management tools include HashiCorp’s Vault and Okta’s Advanced Server Access.
"It's not just a problem for SaaS providers [or CircleCI]," Norton said. "All other CI/CD providers are equally likely to have a similar incident. It’s a matter of when, not if, at this point."
Overall, the headaches for SecOps pros are mounting, especially over the last two months. CircleCI's Jan. 4 advisory was issued just as a series of powerful storms hit Northern California this week, knocking out power and otherwise compromising some SecOps teams' ability to access their systems. It also comes on the heels of other high-profile IT vendor security incidents, including a breach at ChatOps that vendor Slack reported on Dec 31.
"It's been a busy few months with Rackspace, LastPass, CircleCI and Slack," Wright said.
Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
Regular secrets rotation is a SecOps best practice, but enterprises still report credentials are being stolen from code repositories and used by attackers.
