Buildkite: CI/CD pipelines still a job for app developers

Buildkite expands delivery platform

This week, Buildkite mounted a campaign to expand its territory in the developer platform market with a new package registry, test automation features and hosted mobile development service. The Package Registry, based on the company's 2023 acquisition of Packagecloud, centralizes open source artifact management for different programming languages, including self-hosted storage for package data and software supply chain security features such as provenance attestation.

Test Engine assesses the effectiveness of continuous integration tests and recommends which ones should be deleted or refactored as unreliable, or flaky, including automatically assigning test remediation to specific developer teams. The Buildkite Mobile Delivery Cloud consists of a preconfigured infrastructure of Mac M2 hardware with optimized caching where customers can test MacOS and iOS code.

"Our hybrid model and our self-hosted runner approach means that you can deploy Buildkite pipelines into unique hardware, like phones and cars," Pitt said. "[But hosted] Mac compute [for backend iOS app dependencies] is still not a solved problem -- some hyperscalers do offer [it], but it's often highly virtualized and highly difficult to wield, and there's a high barrier of price to use those systems."

All three new features will be part of Buildkite's new Scale-Out Delivery Platform, generally available as of this week and priced starting at $30 per active user per month for the Pro edition; a Custom edition is also available. Buildkite's CI/CD pipeline tools were previously priced starting at $19 per active user per month. Existing customers that want to stick with the previous version of the product aren't required to upgrade, Pitt said.

Some customers might already be using other package managers or test automation tools, Pitt acknowledged, but in many cases, "You'd be surprised -- with Test Engine, there's not a lot of [direct commercial competition] floating around out there already. … We have seen a lot of teams build these things internally, and I call them 'promotionware.'"

Buildkite users solve tricky CI/CD issues

So far, Buildkite's typical users are web-based and tech-focused, including e-commerce service provider Shopify, hospitality orchestrator Airbnb and rideshare company Uber. In addition to self-hosted runners, Buildkite's architecture supports high levels of concurrency, so that thousands of developers can make thousands of changes per day in a relatively short period of time.

"It's kind of something you might graduate to, but maybe not start with," said Jim Mercer, an analyst at IDC. "It definitely would appeal more to the enterprise level where you have multiple very large applications … although anytime you're doing a build, it's a little bit of a developer wait state … everybody's looking to go faster."

Shopify is a big influence for Persona, an online identity verification service provider in San Francisco, since its core application is also a Ruby on Rails monolith. That helped get Buildkite onto Persona's evaluation shortlist two years ago, along with CircleCI, Jenkins, GitHub Actions and incumbent vendor Atlassian Bitbucket, according to Ian Chesal, head of infrastructure for Persona.

"The big thing [that made Buildkite stand out] was unlimited parallelism that … would allow us to basically throw money at inefficiency problems and not have to go through big refactors of our RSpec test suites, or figure out how to do selective test execution, which is not easy in Ruby," Chesal said.

Ultimately, however, switching from Bitbucket to GitHub for source code version control and Buildkite for CI/CD resulted in a 50% cost savings on cloud infrastructure for testing, since Persona could put the self-hosted runners on cheaper spot instances in Google Cloud Platform, while still cutting test suite execution time from more than 20 minutes to fewer than 10.

The company was also an early adopter of Test Engine at the beta stage, Chesal said.

"We had been wanting to use a merge queue on our monolith Rails application for quite some time [because] keeping the main branch clean and deployable at all times is really important to us," he said. "But to adopt a merge queue, you really, really need to have no test flakes, because every test failure [means] it just takes longer and longer to merge your pull requests."

Test Engine lowered the number of flaky tests from 40% to 10% for Persona, according to Chesal. He plans to test the Package Registry next year, as it could cut cloud egress costs the company currently pays to shift artifacts into Buildkite from JFrog.

"That was actually one of the other things that got us looking at alternate CI providers -- Bitbucket was hosted in AWS, and we're all in Google Cloud, and moving artifacts out of our artifact registry into the CI system there was costing us $10,000-plus a month," he said. "Some of that was inefficiencies in caching on the pipeline side and on our part, but some of it was just the hard truth about egress from cloud vendors."

Database-as-a-service provider PlanetScale also looked to Buildkite about two years ago to speed up development on often massive build servers, with as many as 32 or 64 CPUs, according to Nicholas Van Wiggeren, PlanetScale's CTO.

Previously, PlanetScale had used GitHub Actions for CI, and some aspects of that platform remain in use where the build process requires complex coordination between multiple source code repositories.

"We've got a bit of GitHub native stuff in the GitHub Actions side where we would need to really make sure that Buildkite is able to support before we could consolidate," Van Wiggeren said. "It would be cool if we could do stuff like automatically import GitHub Actions [workflows] into Buildkite. That's something I've dreamed about for a really long time."

Buildkite has professional services teams that help with migration from other vendors, but is working on packaged parsers for such conversions, according to a company spokesperson this week.

Van Wiggeren said he's eager to try out Test Engine.

"Half of developers' time is making sure that code is executing successfully, which means running tests, writing tests, adding tests," he said. "Features that let us run those tests as fast as possible, get feedback as fast as possible, get news about failure or success faster -- that is like money back in my pocket as the CTO -- the [build server] hardware is the cheap part when it comes to this whole thing."

Broader enterprise appeal an uphill battle

Buildkite also avoids the term DevOps platform, but this week's additions put it into competition with other CI/CD pipeline vendors that have undergone expansion over the last five years, from Harness to JFrog, according to Mercer.

"I could see [Buildkite] being used by platform engineers to build pipelines for developers," Mercer said. "[Buildkite's leaders are] shying away from the terms DevOps and DevOps platform, but with the expanding capabilities, it feels very much like they're moving in that direction."

Other DevOps platform vendors, such as GitLab and GitHub, can check many functional boxes with their products and have the strong name recognition among business leaders that make purchasing decisions at large companies, Mercer said.

"It definitely has appeal to developers, but then you've got to think, 'OK, who's buying the software?'" Mercer said. "Is it appealing at the enterprise level? That's where you start going up against some of the big guns, and need to [answer], 'Why this as opposed to this huge contract with Microsoft and GitHub?' That's challenging for [Buildkite]."

Beth Pariseau, senior news writer for TechTarget Editorial, is an award-winning veteran of IT journalism covering DevOps. Have a tip? Email her or reach out @PariseauTT.