IT pros revise pipelines for software supply chain security
Software supply chain security has reached an awkward stage for enterprise IT, as platform and security pros grapple with adding upstream tools to existing workflows.
Presenters at KubeCon + CloudNativeCon North America this month have added cloud-native software supply chain security to existing workflows, but best practices and standard toolsets have yet to be established in a scattered and immature market.
Software supply chain security became a hot topic following the SolarWinds breach in 2020, in which bad actors compromised a software delivery pipeline used to build popular IT monitoring software. In the years since, tools including software bills of materials (SBOMs), container image scanning, code and package signing and attestation have emerged with the goal of establishing a trusted chain of custody for software artifacts that consumers can verify digitally. But vendors are just putting together comprehensive software supply chain security products that span from open source code to production runtimes. The cloud-native open source community is also home to multiple projects, but users must evaluate and then apply them to already complex IT environments.
"Understanding and comprehension of what the software supply chain is continues to pose a challenge," said Katie Norton, an analyst at IDC. "Many [organizations] fail to recognize that it is the entire system of producing [software], from conception to delivery of the end product."
In 2023, the industry's focus was on open source security and SBOMs, Norton said.
"In 2024, I have noticed that CI/CD or pipeline security has [been more] recognized as a core component of supply chain security," she said. "Toward the tail end of the year, things like provenance and attestation are coming up more, and I expect that to grow in recognition in 2025."
Yelp designs framework for securing container images
Engineers at Yelp Inc., which runs a crowd-sourced business review app and website, presented in a breakout session at KubeCon about adapting vulnerability management practices to improve the security of container images deployed on the company's internal development platform.
The team chose to focus on containers because that's where most attacks start, said Tom Robinson, a software engineer on Yelp's backend infrastructure team, during the presentation.
"Containers are part of your attack surface, and even though you're thinking about your application at scale, you also have to think about the [smallest] component and what that means," Robinson said.
Robinson and co-presenter Carmen Chow, an infrastructure security software engineer at Yelp, developed a rubric to evaluate the security posture of thousands of individual containers so that those with the lowest score could be blocked from deploying and remediated.
Industry models for container security from Red Hat, the Center for Internet Security and NIST offered guidance, but had to be adapted to Yelp's specific environment, Chow said.
"For example, the CIS Docker Benchmark recommends that we only install the necessary amount of packages into our containers," she said. "But when you have as many containers as we do, each that have a few different requirements, it's hard to [define a] standard for all of these containers."
Yelp engineers came up with their own weighted list of container risk factors, such as the level of access to other systems, the sensitivity of the data processed by the container, and the impact if a container were to be exploited. They combined these factors into an overall score for each container and infrastructure environment. These scores were used to create automated pre-deployment checks for containers in Yelp's CI/CD pipelines, reinforce those checks at runtime and generate audit logs.
The true challenge, Robinson said, is that other organizations can't just re-use Yelp's model -- there isn't one size that fits all.
"It's a little strange because there are plenty of vendors here that are going to come and say, 'Yeah, we can meet all of your needs.' In this particular case, we can't," he said. "Your organization is going to have to think about what your organizational risk is, what your data is, what your containers require, and then use this model in order to define those risks and characteristics."
Adobe and Autodesk put together pipeline pieces
Containers are just one aspect of software supply chain security -- ultimately, a sound software supply chain must be secured from beginning to end, with a software product running in production. More industry benchmarks, such as Supply-chain Levels for Software Artifacts (SLSA), the Open Source Security Foundation's Secure Software Factory and NIST's Secure Software Development Framework, have been developed over the last four years to guide the process.
One key aspect of these secure software development guidelines is a process called attestation. Attestation uses metadata about a software artifact such as how it was built and where it came from -- described with the term provenance -- that is authenticated using a cryptographic signature to ensure it hasn't been tampered with.
As with container security benchmarks, each enterprise must find a way to apply such frameworks to an existing environment, which can be a highly complex undertaking, according to co-presenters at another KubeCon breakout session. They acknowledged that, when viewed together, the many moving parts of end-to-end attestation can be overwhelmingly complex.
"There's a lot happening," said Vikram Sethi, principal scientist at Adobe, ahead of the presentation showcasing a reference architecture for software supply chain security. "Each of these concepts might be a presentation in itself."
For Sethi's co-presenter, a Cloud Native Computing Foundation (CNCF) incubating project, in-toto, has helped. In-toto provides a framework and set of mechanisms for attestation throughout the software delivery process.
"The in-toto project offers up a specification of metadata [for] attestations … and [there's] a nice CLI tool called Witness, which helps us generate those during the [continuous integration] build process," said Jesse Sanford, software architect at design software maker Autodesk in San Francisco, during the presentation. Autodesk uses a sister project to Witness called Archivista as a data lake to store and retrieve attestation metadata, he said.
While these tools facilitate attestation, they could add more polish in future releases, Sanford said in a followup interview with TechTarget Editorial.
"What is most lacking is a consistent and easy way to wrap new [data types] into Witness," he said. "They have plans to make it pluggable, but right now, you have to add the Go structs and associated CRUD [create, read, update and delete] logic to the binary for them one at a time."
Upstream tools interplay and overlap
Witness and Archivista are open source tools contributed to CNCF by TestifySec, a software supply chain security and compliance vendor. Another CNCF project, Notary, which is backed by Microsoft, has a CLI signing tool called Notation. The most popular software signing and attestation CNCF project, Sigstore, uses the Cosign CLI utility.
Teasing apart the differences between tools available for each stage of the software supply chain can be tricky, especially because recent project updates have erased or made most of those differences academic.
One KubeCon presenter said he chose Notation because it was easiest for his team to use with an existing public key infrastructure and certificate authority in HashiCorp Vault.
"We already have jobs that run before a [Kubernetes] cluster is initialized, upgraded or changed, in general, which talk to Vault, get certificates and distribute them to our [cluster] nodes. … It's very trivial for us to … just add Notation CLI [labels] to every node," said Tjark Rasche, cloud software engineer at Mercedes-Benz subsidiary Mercedes-Benz Tech Innovation based in Germany.
Cryptographically signing the Kubernetes artifacts that run on cluster nodes is the first step toward end-to-end attestation for the company, which plans to add attestation and provenance verification in the future, Rasche said.
"In the dream scenario, a developer would push their containers, ideally they already would have signed their commits … and then in the pipeline, after it's built, [the artifact] should be signed," Rasche said. "But let's face it, we're not living in an ideal world."
Still, even an incremental step toward attestation has significantly improved the security of the company's infrastructure software deployments, Rasche said.
"Already, the attack surface has gotten way smaller," he said. "Our [platform] components would be by far the most attractive target because if you would be able to take over the supply chain of the platform team, you could take over all 1,000 clusters."
Enterprise market has yet to coalesce
Some of the fervor that accompanied the initial rush to secure the software supply chain following the SolarWinds breach has faded in the last two years. SBOMs were missing from CISA's self-attestation form for federal software suppliers finalized earlier this year, despite a presidential executive order in 2021 that called for their use by federal agencies.
Meanwhile, vendors that staked their livelihoods on that wave of interest in software supply chain security are now looking for new ways to gain traction. Chainguard, for example, continues to rely on attestation and signatures based on Sigstore and generate build-time SBOMs for its minimalist Chainguard Images, but it has pivoted to focus more on those images rather than its original goal of building a platform to secure the entire supply chain.
"Overall, I see signatures and attestations as a key part of a secure supply chain, which is why we've integrated them into our images product," according to Chainguard co-founder and CEO Dan Lorenc in an interview this week. "But unfortunately, adoption and demand from end users is still lagging behind."
TestifySec's founder and CEO Cole Kennedy said during a KubeCon interview that he expects consolidation between software supply chain security vendors and larger enterprise compliance and risk management vendors, as regulatory compliance is still the strongest incentive to implement software supply chain security practices.
"Developers can push 100 times a day; compliance teams, maybe they can do a package once every 30 days," Kennedy said. "There's a lot of friction happening there, and … platforms that solve some of the issues with shifting compliance left are going to win, while those that are just providing security are going to struggle."
Interest in software supply chain security among respondents to recent market research surveys remains tepid, according to IDC's Norton. Among more than 350 respondents to IDC's "DevSecOps and Software Supply Chain Security Survey, 2024," 26% of organizations indicated they are digitally signing binaries or containers, and 21% were generating provenance metadata.
"Organizations are often looking to their existing application security and CNAPP [cloud-native application protection platform] tools to provide them with integrated supply chain capabilities," Norton said. "And as these platforms continue to broaden their scope, it further entrenches them in enterprises, making it harder for startups trying to disrupt."
Beth Pariseau, senior news writer for TechTarget Editorial, is an award-winning veteran of IT journalism covering DevOps. Have a tip? Email her or reach out @PariseauTT.
