Enterprises' production experience with GitOps and tools updates have helped early adopters overcome a common operational obstacle: configuration management.
Kubernetes and the applications deployed via the container orchestration framework are made up of manifests written in human-readable language formats such as YAML and JSON. GitOps, an approach to application and infrastructure deployments that continuously reconciles the state of running systems with a desired state expressed in code, relies entirely on these declarative manifests. Additional manifest files are also required to manage the configuration of secondary services associated with applications, such as Istio service mesh or Prometheus monitoring.
Configuration management via YAML or JSON files therefore lies at the heart of GitOps but often becomes unwieldy at scale. In large organizations, where thousands of developers make multiple code changes daily, these files can quickly number in the millions. Each of the most popular Kubernetes manifest creation tools -- Helm charts, Kustomize overlays and a Jsonnet-based tool called Tanka -- also comes with its own tradeoffs in terms of simplicity and flexibility for developers.
However, as companies such as restauranteur Chick-fil-A gain experience with GitOps, platform teams have learned to cope with that sprawl. One way Chick-fil-A engineers reduced configuration management confusion was by creating base manifests, stored in a separate set of repositories, which developers can reuse rather than create their own custom configuration file for each deployment.
"The base manifests are merged with any app-specific overlays found in the app repo [at deployment time]," said Christopher Lane, an enterprise architect at Chick-fil-A, in a GitOpsCon presentation this week. "Developers are free to patch anything in the standard manifest, but we offer sane defaults that work for most use cases."
Once the base manifests and custom overlays are merged, they're committed to another Git repository called Atlas in Chick-fil-A's application platform, which the Argo CD GitOps controller monitors and reconciles with the state of production systems.
"This provides kind of an easy on-ramp for developers to get started quickly, increase their knowledge incrementally and change things as they need to without having to wade through a gigantic pile of YAML," Lane said.
Chick-fil-A reps present GitOps process.
GitOps tools polish multi-cluster management
For coffee retailer Starbucks, version 2 of the Flux GitOps controller, which reached stability in June, has eased GitOps configuration management because it improved multi-repo support; a single instance of Flux can now connect to multiple Git repositories and centrally manage updates to multiple Kubernetes clusters.
It's a problem we're constantly working on solving because scale is really important when we're looking at deploying clusters out to hundreds or thousands of [store] locations.
David Lewis Lead SRE, Starbucks
"You can define a single Git source for all of your clusters to manage the configuration at scale, and just provide individual overlays that will perform specific configurations for a cluster," said David Lewis, lead site reliability engineer for the Seattle-based company, of multi-repo support in Flux2, in an interview. "It's a problem we're constantly working on solving because scale is really important when we're looking at deploying clusters out to hundreds or thousands of [store] locations."
Argo CD version 2, released in April, added a new feature called ApplicationSets, which makes it easier to manage large numbers of applications deployed among multiple Kuberntes clusters as a single logical unit. This includes a new Git file generator that automatically deploys configuration data to specific target clusters, streamlining configuration management.
"In a new, large internal multi-tenancy cluster that I'm putting together, the strategy is to use Argo CD ApplicationSets ... to create multiple namespaces for the multiple clusters," said Chick-fil-A's Crane. "Right now, we have an Argo CD [instance] per cluster ... but as we move forward, we see Argo as a control plane that we want to have separate from the cluster itself."
GitOps configuration management can still improve
As with many aspects of the nascent GitOps concept, configuration management is not a completely settled issue. More must be done to help developers and platform operators enhance this process, GitOpsCon speakers said.
For example, the Argo CD GUI can show developers the YAML or JSON behind their configuration settings, which they can cut and paste into Git repos. But ideally, the UI would allow them to skip that step, Crane said.
"How nice would it be if [the system] was able to load that from Git as the data store when you went to that menu, and when you hit 'save,' [create] a pull request that can be reviewed?" he said. "That would enable those users who are more UI- or visual-based, but not box out those who do need to go edit something more directly."
For Starbucks' Lewis, maintaining base configuration manifests on behalf of developers can simplify a GitOps deployment, but updating those base manifests can be tricky for operators to do while maintaining proper code quality.
"All of those changes could get immediately propagated to hundreds of production clusters, but you need to test all of those changes first," he said. "There are a couple of possible solutions for this, such as versioning your base configuration, but they do require a bit of extra work."
Another GitOps early adopter enterprise, Fidelity Investments, has developed its own solution to managing configuration files with an open source project. First released last year, Kraan coordinates Kubernetes add-on layers, including Helm charts through Flux's HelmRelease API.
"Kraan comes [in] to provide dependency management between HelmReleases or groups of HelmReleases," said Adrian Vacaru, a software engineer at Fidelity, in an online interview. "If you have a complex suite of applications that need to be deployed at once, Kraan will make sure they all come up in the correct order."
Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
Dig Deeper on Systems automation and orchestration
