Kubernetes complicates Cloud Foundry platform development
The Cloud Foundry community has begun to port its developer interface onto Kubernetes infrastructure, but there are multiple approaches to sort out amid the transition.
Cloud Foundry has entered an unprecedented phase of its development, as upstream maintainers look to swap out their platform's infrastructure engine without slowing down the enterprise developers who rely on it.
This process began in earnest last year, following years of debate about whether the change of course in cloud platform development was necessary. Portions of the Cloud Foundry community had begun to bend toward Kubernetes with Project Kubo in 2017, an early effort to join the two platforms by Pivotal, which is now part of VMware. But as recently as 2018, the wider Cloud Foundry user base was still undecided on whether to replace the infrastructure that had already been developed for the platform with Kubernetes equivalents.
Last spring, the conversation began to shift, with early releases of Project Eirini presented at a sparsely attended Cloud Foundry Summit in Philadelphia. A little over a year later, with a new Cloud Foundry Foundation director at the helm, the 2020 virtual edition of the Summit -- de rigeur amid a global pandemic -- refocused on cloud platform development within the community, rather than on enterprise end users.
That community is now operating based on a consensus that Kubernetes is the way to go, according to Chip Childers, who took over for former Foundation director Abby Kearns in early April.
"Our community is realigned around [reimagining] the architecture of a lot of subsystem components to take advantage of Kubernetes," Childers said in a recent interview. "Everything is moving in the same direction."
Stephen O'Grady
Commercial Kubernetes platforms such as IBM Red Hat OpenShift and cloud-provider hosted Kubernetes services such as GKE already offer an optimized developer experience on Kubernetes. But the Cloud Foundry community still has a chance to add value based on its years of experience in cloud platform development, said Stephen O'Grady, principal analyst and co-founder at RedMonk in Portland, Maine.
The question is no longer whether or even how to drive Cloud Foundry and Kubernetes together -- instead, it's how Cloud Foundry can provide abstractions above and around Kubernetes to improve the overall developer and operator experience.
Stephen O'GradyPrincipal analyst and co-founder, RedMonk
"Cloud Foundry has been offering that type of optimized experience for a long time now," O'Grady said. "The question is no longer whether or even how to drive Cloud Foundry and Kubernetes together -- instead, it's how Cloud Foundry can provide abstractions above and around Kubernetes to improve the overall developer and operator experience."
Among the Foundation's recent efforts toward that goal was the GA release of version 7 of the Cloud Foundry command line interface (CLI) during the Summit last week. The new version, which could automatically be inherited by downstream commercial versions of Cloud Foundry such as VMware/Pivotal's PKS and Tanzu products, eases container-friendly rolling application deployments. It also supports more nuanced multi-step application rollouts, as well as sidecar processes important to support service mesh.
Cloud Foundry Foundation is now united around the goal of steering its cloud platform development toward Kubernetes, but the state of that work remains somewhat fragmented as of this year's Summit.
Last year, the event focused on the release of Project Eirini, which allows developers using the Cloud Foundry Application Runtime (CFAR) to choose between Kubernetes and Cloud Foundry's own Diego and Garden utilities for container management and scheduling. Eirini has since been combined with Project Quarks, which packages CFAR into containers rather than its original virtual machine format.
Two more projects have also begun in the last year to facilitate automatic deployment of the containerized Cloud Foundry platform onto Kubernetes infrastructure. The more mature of these efforts is KubeCF, a project spearheaded by SUSE that reached a production-ready version 1.0 and was released to incubation in the Cloud Foundry Foundation in May. KubeCF uses a Cloud Foundry Operator developed within the Cloud Native Computing Foundation, and still relies heavily on BOSH, Cloud Foundry's core VM-based infrastructure provisioning and orchestration tool.
The CF-for-K8s project was also released this spring in version 0.1.0, ready only for sandbox use. This project eliminates BOSH and provides interfaces directly between CFAR and components redesigned for use with Kubernetes, such as kpack, a container packaging utility designed for Kubernetes, where KubeCF still uses the existing Cloud Foundry buildpacks. CF-for-K8s similarly swaps out Cloud Foundry's Go router for the Istio service mesh, and a new kapp CLI for the previous BOSH CLI.
"These projects are not in opposition," wrote Troy Topnik, a senior product manager at SUSE, in a Cloud Foundry blog announcing CF-for-K8s. "They are convergent."
Topnik's post, and presentations at this year's Summit, positioned KubeCF as an easier on-ramp to Kubernetes integration for existing Cloud Foundry shops, while CF-for-K8s represents a bigger, longer-term leap away from BOSH.
Still, Cloud Foundry project maintainers conceded in a panel session at this year's Summit that ultimately, two separate cloud platform development projects with similar goals is less than ideal.
"Someone coming from the BOSH world is going to have a very soft landing spot in KubeCF … because you would not have to learn all the Kubernetes concepts," said Simon Moser, senior technical staff member at IBM, which will incorporate KubeCF within its IBM Cloud Foundry service. "Whereas someone starting on the greenfield or … looking to use all the very latest Kubernetes concepts, I think CF-for-K8s is going to be the more natural match.
"Having said that," Moser added during the panel discussion, "I'm really, really hoping that when we get together for this panel in a year again, that those two projects will be merged."
The long goodbye to BOSH
As the community's cloud platform development priorities shift, several Summit attendees in online Q&A sessions expressed concerns about the long-term future of BOSH.
"BOSH has the advantage of providing well-isolated VMs [and] we have customers that do not want to containerize their data service instances," said Julian Fischer, CEO at Anynines GmbH, a German company that provides IT managed services and platform design based on Cloud Foundry, to one such questioner. "When this problem has been solved sufficiently [it is possible] BOSH may have to transform or it will become irrelevant over time. I hope for the first of the two options."
Whether BOSH evolves or fades away, it will take quite some time for Kubernetes to match the utility for easy, standardized multi-cloud provisioning, Fischer said in a follow-up interview this week.
"We still sell BOSH environments and it will be there for years," he said. "If I were building a new environment today, I might still choose VMs, because they're very well-isolated and work like a charm at scale."
Efforts are afoot in the Kubernetes community to fulfill the container orchestrator's cloud-agnostic container management promise, but those efforts, such as Cluster API, are still early-stage works in progress.
BOSH creates virtual machines, virtual disks and some network infrastructure, but users still need infrastructure-as-code utilities such as Terraform to flesh out a new infrastructure environment fully. This takes time and effort, but once established, BOSH uses its Cloud Provider Interface (CPI) integrations with cloud infrastructure APIs to stamp out subsequent deployments much more quickly and reliably than anything yet available for Kubernetes, Fischer said.
Another difference between BOSH CPIs and Cluster API is that the latter also incorporates the infrastructure-as-code layer -- for example, Cluster API for AWS generates CloudFormation templates. This additional layer of abstraction raises the potential for "ugly heterogeneities" in Kubernetes multi-cloud deployments, Fischer said.
In some ways, Kubernetes is repeating Cloud Foundry's platform development, but Fischer doesn't see Kubernetes as "the enemy," he said.
"Welcome to the technology space," Fischer said. "We are in the fashion industry, and Kubernetes is the new black."
Stephen O'Grady
