Deutsche Telekom preps Kubernetes 5G core with GitOps

GitOps fuels 5G network speed

DT first began to plan core telecom services that would run on its Kubernetes platform, which it calls "Das Schiff," or "The Ship," in 2019. In part, this came about because server virtualization vendors and the Kubernetes open source community had improved support for network functions virtualization (NFV), a set of software frameworks that allow virtual machines to support high-performance network devices, such as routers and load balancers, that previously required specialized hardware. DT also had to wait for telecom software vendors to update their apps to support VMs and containers.

By late 2019, DT had moved conventional IT applications onto Das Schiff, but hosting 5G SA there presented a unique challenge beyond NFV's ability to emulate hardware-based performance. The company's software applications and software-defined infrastructure updates, too, had to move faster.

There would be no [other] way to control quality ... and immediately react to a customer demand.

"One second in 5G is like forever," Sewera said. "It's a huge difference compared to any other IT application -- not only from the perspective of [sensitivity to] outages, but especially if you think about the runtime integration where thousands of nodes are connected and you need to reply in milliseconds."

As it built Das Schiff, DT's infrastructure team had begun to work with the open source Flux CD GitOps tool to manage its Kubernetes clusters, which now number in the hundreds, with an eye toward an eventual fleet of thousands.

When Sewera's team started planning to move 5G SA to Das Schiff in 2020, it found GitOps to be an indispensable part of making that leap. While very infrequent manual intervention is required from DevOps and IT infrastructure teams, a GitOps workflow means Das Schiff is largely self-updating for the entire lifecycle of each application, automatically syncing production environment resources and configuration with changes expressed in code, and less prone to the human errors that come with manual testing and deployment.

"You're doing a very small increment of change on everything, and there is a concept which I call nonstop acceptance testing, which means that there is no longer a fixed point of time where your app is frozen [for testing]," Sewera said. "There would be no [other] way to control quality ... and immediately react to a customer demand."

DT's road to Kubernetes 5G core

In 2019, DT's infrastructure team began to consider various commercial Kubernetes management platforms, as well as a self-managed environment based on upstream open source. After it decided to go with the latter option, the infrastructure team also began to experiment with GitOps, engaging Flux CD commercial supporter Weaveworks for support and consulting around how to implement the tool at high scale.

"They were early adopters of some technologies we currently use, like Cluster API, in their own Kubernetes platform for enterprise," said Vuk Gojnic, squad leader for DT's Kubernetes engine. "We had other contestants for that kind of role that were heavily using ... [HashiCorp] Terraforming. For our small team at that time, it was not looking like something that would scale."

Using Flux, DT now manages some 200 Kubernetes clusters with just 10 full-time engineers and plans to scale to thousands of clusters without adding more than one or two more members to the infrastructure team.

DT's investment fund, Telekom Investment Pool, later invested in Weaveworks, along with other investors including Ericsson Ventures and Orange Ventures, in a $36 million series C funding round in 2020.

DT's work with Weaveworks helped it weather the transition between Flux version 1 and Flux version 2, first released in 2020, which added sought-after support for multi-tenancy, but represented a ground-up refactoring for the utility.

"We had a lot of interactions with the [Weaveworks] engineering team and gave them a lot of feedback that got built into the upstream product -- a number of things that are specific to how [we] manage application deployment profiles," Gojnic said.

Weaveworks incorporated Flux version 2 into its Weave GitOps Enterprise product, launched in June and updated this week in version 2021.11, which added an application delivery UI, reusable infrastructure component profiles, support for reproducible platforms as code and a multicluster management interface.

DT was an early user of Flux version 2, but signed contracts with Weaveworks for Flux technical support before the release of Weave GitOps Enterprise, Gojnic said, and may consider the commercial platform in the future.

"We are running a custom stack that we have support for," Gojnic said. "It's not an off-the-shelf product, but we are also hoping to make this use case [work]."

Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.