Behind the scenes, Spotify's Backstage a work in progress
Spotify-created Backstage, now a CNCF project, is popular as enterprises add developer portals to internal platforms, but setup is hard, and getting dev buy-in can be even harder.
CHICAGO -- Spotify's Backstage developer portal framework has quadrupled its user base in 2023, but usage is only the first step.
The project's goal is to "restore order" to users' infrastructure by providing a centralized self-service software catalog for developers that covers all aspects of enterprise software development and deployment, according to its website. Donated to the Cloud Native Computing Foundation (CNCF) in 2020, the project's core features have reached maturity at a time when many enterprises want to add developer self-service portals to internal development platforms according to the platform engineering trend.
Thus, Spotify's Backstage has amassed more than 2,300 users and more than 2,000 contributors from vendors including VMware, Red Hat, Cisco and Harness, and customers such as online retailer Zalando and American Airlines. VMware, Red Hat and Harness have integrated Backstage into developer platform products.
But most of that growing user base has yet to achieve widespread developer adoption for Backstage within their companies. Spotify officials have estimated publicly that the average Backstage adoption rate is 10%. To break out of those small pockets of usage within enterprises, the project must become easier for platform engineers to set up and update, according to Spotify project managers at this week's BackstageCon event.
The main theme of this year is simply ease of use. We want to lower the threshold for people to get started with Backstage, we want to make it easier to contribute, [and] we want to make it easier to maintain.
Helen GreulHead of engineering for Backstage, Spotify
"Inevitably, we need to address some of the scaling challenges that come with growth," said Helen Greul, head of engineering for Backstage at Spotify, during opening remarks at the conference. "The main theme of this year is simply ease of use. We want to lower the threshold for people to get started with Backstage, we want to make it easier to contribute, [and] we want to make it easier to maintain."
Among the recent updates to Spotify's Backstage is a Quickstart utility in private beta that reduces some 70 setup steps to between three and five. This week, Spotify also launched a marketplace for Backstage, which houses a set of vetted plugins from six vendor partners, including Red Hat and VMware. Plugins are among the main components of Backstage and are used to integrate third-party products and services into users' software catalogs.
Spotify rolled out a subscription bundle of plugins last December that covers common usage patterns for Backstage, including compliance, access control, employee education and satisfaction, and usage metrics. But until now, it was difficult for users to evaluate more than 150 third-party plugins and determine if they were secure and well maintained. The marketplace is Spotify's answer to that adoption challenge.
The new marketplace is a good place to get started, but other roadmap items will be crucial once organizations get past the initial phase of adoption, according to a panel of experienced Backstage users at BackstageCon.
Spotify product manager Meg Watson introduces the Spotify Marketplace for Backstage and presents on project roadmap goals at BackstageCon.
"I really like that we now have a marketplace, because that just makes adoption easier," said Kasper Nissen, lead platform architect at Lunar, a digital financial services company based in Denmark, during a Q&A portion of a BackstageCon panel session. "But [eventually], you don't just pull things down from whatever open source shelf is out there -- because you probably need to tailor them to your organization, to your developers and your unique workflows."
That's where other fundamental improvements come in to Backstage frontend and backend packages that Spotify has planned for early 2024.
Frontend packages run an organization's custom-developed apps and underpin the self-service catalog interface for developers. Backend packages are used to run Backstage itself. Both also contain plugins and extensions that wire plugins together to form an application.
Until recently, Backstage's frontend packages had a more mature set of services that developer productivity teams could use to create applications in a catalog; similar backend services are now on the way, according to the Spotify roadmap. These backend services will make it simpler for platform engineers to support utilities such as logging and database access so that each doesn't require re-implementation from scratch.
"As our Backstage instance has been scaling out and we're starting to get internal contributors to help scale it out, we're pretty excited for the new backend system and the new composability features to really help us [improve] that ability to contribute easily," said panelist Alec Jacobs, staff software engineer at Twilio.
Frontend packages are due for their own face-lift under a proposal called declarative integration, which Spotify presenters said is due for an early release by the end of the year. Declarative integration is meant to remove the need to write TypeScript code to integrate plugins, and to make the code required for that integration less verbose.
Red Hat shores up Backstage security, adds dynamic plugins
Backstage partner Red Hat also released updates this week meant to address the adoption challenges blocking the project from more widespread use -- most notably, authorization and permissions policy management -- according to a BackstageCon presentation by Balaji Sivasubramanian, head of product management for developer tools at Red Hat.
"The Backstage permission framework requires creating and maintaining an RBAC [role-based access control] plugin and all its significant reporting and ongoing maintenance work," Sivasubramanian said. "That's not a very scalable process for enterprises."
This week, Red Hat launched its own free and open source RBAC plugin, which is configurable via a configuration file or Red Hat's Backstage UI, and is itself manageable according to enterprise policy controls.
Red Hat also contributed an experimental dynamic backend plugin utility to Backstage version 1.18, released in September, which allows Backstage users to add or update plugins without having to modify source code, rebuild and restart the system. Spotify's declarative integration project will help pave the way for built-in dynamic frontend plugin support, presenters said. Spotify also has a CNCF security audit for Backstage on its roadmap.
Backstage users chip away at developer buy-in
Technical updates will help existing Backstage users broaden adoption within their organization, but the blockers to that adoption are as organizational as they are technical.
For example, an informal survey of 20 Backstage user companies conducted by David Tuite, founder and CEO at Backstage SaaS provider Roadie, showed that companies with existing application catalogs they could import into Backstage via a custom processor were more successful in that effort than companies that built software catalogs using YAML files.
That question regarding software catalogs revealed more of a cultural challenge than an issue with how Spotify's Backstage handles YAML files, according to Lunar's Nissen, who was among the respondents surveyed by Tuite and who used a custom processor.
"What many adopters are probably struggling with is that they have scattered a lot of stuff around in different repositories, they use different languages, and they're probably deployed in different ways, so they don't have a standard way to describe a service," he said in an interview at BackstageCon this week. "You need all that data to have a successful catalog."
Enterprise users of Spotify's Backstage in a panel session at BackstageCon, left to right: Alec Jacobs, Twilio; Kasper Nissen, Lunar; Poonam Garg, U.S. Bank; and Guillermo Manzo, Expedia.
It also takes time to get all the stakeholders within an organization aligned around a central catalog. At Lunar, for example, the focus so far has mainly been on setting up the Backstage backend, and the company is still working on a frontend catalog UI for developers.
In highly regulated environments such as the financial services industry, fears around security and compliance risks have stymied Backstage adoption for companies such as U.S. Bank, according to one of the BackstageCon panelists.
"We have everyday calls with auditors, and risk and compliance, and security -- they keep on us," said panelist Poonam Garg, vice president and engineering lead at U.S. Bank. "They are double-checking with us for everything ... and we are also educating them at the same time."
Demonstrating the value of Backstage in developer time saved and software deployment velocity is crucial to selling the portal as an internal product, early adopters said. But in the broader market, Backstage will also be limited by its inherent focus on cloud-native apps, as most Spotify deployments require containerization, according to one analyst.
"Unless you have those chops or use all modern and cloud infrastructure, it is still a bit of a science project," said Rob Strechay, lead analyst for enterprise tech media company TheCube. "The problem becomes who supports it at scale and who certifies it for regulated industries."
Beth Pariseau, senior news writer at TechTarget Editorial, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
