Weissblick - stock.adobe.com
Kubernetes security opens a new frontier: Multi-tenancy
Enterprises have grown used to managing Kubernetes in production for some of their workloads, but broadening those deployments can bring multi-tenant security challenges.
SAN DIEGO -- As enterprises expand production container deployments, a new Kubernetes security challenge has emerged: multi-tenancy.
Among the many challenges with Multi-tenancy in general is that it is not easy to define, and few IT pros agree on a single definition or architectural approach. Broadly speaking, however, multi-tenancy occurs when multiple projects, teams or tenants, share a centralized IT infrastructure, but remain logically isolated from one another.
Kubernetes multi-tenancy also adds multilayered complexity to an already complex Kubernetes security picture, and demands that IT pros wire together a stack of third-party and, at times, homegrown tools on top of the core Kubernetes framework.
This is because core upstream Kubernetes security features are limited to service accounts for operations such as role-based access control -- the platform expects authentication and authorization data to come from an external source. Kubernetes namespaces also don't offer especially granular or layered isolation by default. Typically, each namespace corresponds to one tenant, whether that tenant is defined as an application, a project or a service.
"To build logical isolation, you have to add a bunch of components on top of Kubernetes," said Karl Isenberg, tech lead manager at Cruise Automation, a self-driving car service in San Francisco, in a presentation about Kubernetes multi-tenancy here at KubeCon + CloudNativeCon North America 2019 this week. "Once you have Kubernetes, Kubernetes alone is not enough."
However, Isenberg and other presenters here said Kubernetes multi-tenancy can have significant advantages if done right. Cruise, for example, runs very large Kubernetes clusters, with up to 1,000 nodes, shared by thousands of employees, teams, projects and some customers. Kubernetes multi-tenancy means more highly efficient clusters and cost savings on data center hardware and cloud infrastructure.
"Lower operational costs is another [advantage] -- if you're starting up a platform operations team with five people, you may not be able to manage five [separate] clusters," Isenberg added. "We [also] wanted to make our investments in focused areas, so that they applied to as many tenants as possible."
Multi-tenant Kubernetes security an ad hoc practice for now
The good news for enterprises that want to achieve Kubernetes multi-tenancy securely is that there are a plethora of third-party tools they can use to do it, some of which are sold by vendors, and others open sourced by firms with Kubernetes development experience, including Cruise and Yahoo Media.
Duke Energy Corporation, for example, has a 60-node Kubernetes cluster in production that's stretched across three on-premises data centers and shared by 100 web applications so far. The platform is comprised of several vendors' products, from Diamanti hyper-converged infrastructure to Aqua Security Software's container firewall, which logically isolates tenants from one another at a granular level that accounts for the ephemeral nature of container infrastructure.
"We don't want production to talk to anyone [outside of it]," said Ritu Sharma, senior IT architect at the energy holding company in Charlotte, N.C., in a presentation at KubeSec Enterprise Summit, an event co-located with KubeCon this week. "That was the first question that came to mind -- how to manage cybersecurity when containers can connect service-to-service within a cluster."
Some Kubernetes multi-tenancy early adopters also lean on cloud service providers such as Google Kubernetes Engine (GKE) to take on parts of the Kubernetes security burden. GKE can encrypt secrets in the etcd data store, which became available in Kubernetes 1.13, but isn't enabled by default, according to a KubeSec presentation by Mike Ruth, one of Cruise's staff security engineers.
Google also offers Workload Identity, which matches up GCP identity and access management with Kubernetes service accounts so that users don't have to manage Kubernetes secrets or Google Cloud IAM service account keys themselves. Kubernetes SIG-Auth looks to modernize how Kubernetes security tokens are handled by default upstream to smooth Kubernetes secrets management across all clouds, but has run into snags with the migration process.
In the meantime, Verizon's Yahoo Media has donated a project called Athenz to open source, which handles multiple aspects of authentication and authorization in its on-premises Kubernetes environments, including automatic secrets rotation, expiration and limited-audience policies for intracluster communication similar to those offered by GKE's Workload Identity. Cruise also created a similar open source tool called RBACSync, along with Daytona, a tool that fetches secrets from HashiCorp Vault, which Cruise uses to store secrets instead of in etcd, and injects them into running applications, and k-rail for workload policy enforcement.
Kubernetes Multi-Tenancy Working Group explores standards
While early adopters have plowed ahead with an amalgamation of third-party and homegrown tools, some users in highly regulated environments look to upstream Kubernetes projects to flesh out more standardized Kubernetes multi-tenancy options.
For example, investment banking company HSBC can use Google's Anthos Config Management (ACM) to create hierarchical, or nested, namespaces, which make for more highly granular access control mechanisms in a multi-tenant environment, and simplifies their management by automatically propagating shared policies between them. However, the company is following the work of a Kubernetes Multi-Tenancy Working Group established in early 2018 in the hopes it will introduce free open source utilities compatible with multiple public clouds.
"If I want to use ACM in AWS, the Anthos license isn't cheap," said Scott Surovich, global container engineering lead at HSBC, in an interview after a presentation here. Anthos also requires VMware server virtualization, and hierarchical namespaces available at the Kubernetes layer could offer Kubernetes multi-tenancy on bare metal, reducing the layers of abstraction and potentially improving performance for HSBC.
Homegrown tools for multi-tenant Kubernetes security won't fly in HSBC's highly regulated environment, either, Surovich said.
"I need to prove I have escalation options for support," he said. "Saying, 'I wrote that' isn't acceptable."
So far, the working group has two incubation projects that create custom resource definitions -- essentially, plugins -- that support hierarchical namespaces and virtual clusters that create self-service Kubernetes API Servers for each tenant. The working group has also created working definitions of the types of multi-tenancy and begun to define a set of reference architectures.
The working group is also considering certification of multi-tenant Kubernetes security and management tools, as well as benchmark testing and evaluation of such tools, said Sanjeev Rampal, a Cisco principal engineer and co-chair of the group.
