GitOps can simplify some aspects of Kubernetes deployments, but configuration management isn't one of them, according to enterprise practitioners.
GitOps, in which a Git code repository acts as the central control mechanism for automated application and infrastructure updates, has become increasingly popular among Kubernetes users.
The container orchestration platform, managed through YAML, lends itself to the GitOps method of declarative automation, where the desired end state of applications dictates how they are deployed and updated, rather than the imperative method, which specifies the steps to get to that end state. Such centralized, consistent automation can make it easier for relatively small IT teams to keep large Kubernetes environments updated.
However, Kubernetes YAML also represents the biggest challenge to achieving GitOps at high scale for large enterprises such as Intuit. Most Kubernetes add-ons such as Prometheus monitoring and Istio service mesh also use YAML for configuration; multiply that by 4,000 Intuit developers and 2,500 application services, and the result is some 2.5 million lines of YAML that the company must manage in its GitOps environment.
"Editing YAML on a micro scale of one file has its challenges, but managing all your Kubernetes YAML at a cluster or multiple-cluster scale is also challenging," said Daniel Thomson, software engineer at user authentication vendor Stytch, in a KubeCon virtual presentation this week. Thomson worked as a senior software engineer at Intuit until September.
Intuit mostly eased Kubernetes configuration management with the open source Kustomize open source tool, which abstracts most of the details of Kubernetes deployment from developers. Now, Intuit developers manage only about 25% of the total lines of YAML in the environment, while the rest is handled by the platform team behind the scenes.
However, Intuit also still chose to allow developers access to raw Kubernetes YAML if they desired it, and that makes developer support more challenging for the platform team, said Jesse Suen, principal software engineer at Intuit, in the same KubeCon presentation.
"By making that decision, we also gave our users a lot of loaded foot guns," Suen said. "People who may not be as familiar with Kubernetes can make a lot of mistakes."
GitOps requires new ways of thinking about DevOps automation, including Kubernetes configuration.
Kubernetes configuration management tools' pros and cons
There are several open source Kubernetes configuration management tools available, but each has drawbacks, according to GitOps pros at KubeCon.
Helm, the oldest and most stable of these projects, reached Cloud Native Computing Foundation graduation status in April. It is still used by many Kubernetes add-ons, including version one of the Flux GitOps platform. Helm v3 removed a show-stopping security concern for many enterprises with the deprecation of Tiller, an orchestration controller with elevated privileges that could potentially give attackers access to Kubernetes clusters.
However, Helm is becoming passe for GitOps purposes because, while flexible, it quickly becomes complex -- especially in environments where developers want to make many small configuration changes, Thomson said.
Kustomize became increasingly popular in GitOps shops over the last year, often as an overlay added to Helm, because it can be used to show developers only the configuration fields they care about. Kustomize is also natively supported in the kubectl Kubernetes command-line interface as of version 1.14.
"We use Helm, but we don't deploy Helm directly," said Fabio Giannetti, director of software engineering at Mastercard, during a KubeCon panel discussion. "We use Helm's templating capability and then lay Kustomize over it."
Kustomize also has its drawbacks, according to the Intuit presentation, including the fact that it doesn't use the Helm concept of parameters for flexibility.
"It's very readable and it encourages configuration re-use, and it's mostly flexible," Thomson said. "That said, it's not always immediately intuitive to developers … and the lack of parameterization makes things that should be hard."
Another GitOps configuration management approach that gained momentum over the last year uses programming frameworks, including Java's Jsonnet or a translation layer between Kubernetes YAML and common programming languages offered by tools such as Pulumi. In October, observability vendor Grafana Labs released Tanka, a utility that blends Helm charts with Jsonnet, which combines the deployment speed and ubiquity of Helm charts with the more granular customizability supported by Jsonnet.
It's a combination one GitOps practitioner is eager to try.
"We're starting to use Jsonnet in the form of Tanka [to deploy our observability stack] -- there's a lot of flexibility there that I'm finding to be quite nice, compared to Helm," said Matt Young, principal cloud architect at EverQuote, an online insurance marketplace in Cambridge, Mass., in an interview. "You describe configuration, render JSON and then get YAML from it."
But the Jsonnet approach, like the others, also has significant pitfalls. Applying Jsonnet to Kubernetes isn't a new idea -- a now-defunct project called ksonnet was meant to do exactly that when Heptio engineers introduced it in 2017, but that project went dark after VMware acquired Heptio in 2019 and the industry began to favor Helm.
"You get all the disadvantages of a codebase" with the programming-language approach, Thomson said. "You might have bugs … and it can be a challenge for your developers to figure out, 'how does this chart that we have listed here translate to our final result?'"
GitOps users seek ideal Kubernetes config tool
Ultimately, the industry hasn't found an ideal approach to Kubernetes configuration management, especially for GitOps, Thomson said.
Kubernetes YAML is super powerful and here to stay, but we need a better way of abstracting all the nitty-gritty details of YAML away from users.
Daniel ThomsonSoftware engineer, Stytch
"No matter what you do, certain edge cases or road bumps are going to be a challenge," he said. "We think Kubernetes YAML is super powerful and here to stay, but we need a better way of abstracting all the nitty-gritty details of YAML away from users, because they don't need to know about it."
Thomson said that ideally, he would like to see a UI-assisted configuration management tool that would walk developers through editing YAML files, with insight into the effects of their changes and the ability to enforce organizational standards and best practices. Netflix's CD utility Spinnaker already has such an interface for provisioning EC2 instances, and Thomson said he'd like to see something similar for Kubernetes YAML.
In the meantime, GitOps automation requires caution to ensure that configuration details are rendered correctly, according to KubeCon panelists. Mastercard, for example, checks every deployment with Kubernetes conformance testing tools such as sonobuoy and kubench, Giannetti said during the panel discussion.
"There are things that should just work, but in reality, they sometimes still do need human attention," Young added during the panel discussion. "Exercise caution and don't assume that just because everything's automated, everything's working."
Dig Deeper on Systems automation and orchestration
