SEATTLE -- Kubernetes security issues are under scrutiny among enterprise IT pros as they deploy containers in production.
The breadth of features in the Kubernetes container orchestration platform makes security issues a manifold problem that requires multiple layers of defense to address -- from the reconfiguration of default settings in Docker container images and upstream Kubernetes to cutting-edge security monitoring tools and advanced networking concepts, such as service mesh.
One thing is clear in discussions at KubeCon and KubeSec here this week: Upstream Kubernetes versions and Docker container images are not secure out of the box.
"The [Kubernetes] orchestrator defaults are terrible, so please change them," said Sarah Young, IT security architect at Versent, an Australian IT consulting firm, in a presentation.
For example, the Kubernetes API server listens on port 8080 where no security checks take place by default, and encryption and secrets management within etcd -- the Kubernetes key-value store that maintains cluster state -- remain in development. When used with etcd version 3, Kubernetes version 1.13 -- released in December 2018 -- offers encryption of data at rest. But, as last week's critical Kubernetes security vulnerability revealed, few enterprises can keep up with upgrades to subsequent versions of Kubernetes.
"Kubernetes is just catching up to basic enterprise security best practices," said Tsvi Korren, vice president of product strategy at security monitoring vendor Aqua Security, in a panel discussion. "Encrypting object storage isn't something you should pat yourself on the [back] for."
Security experts said the industry has largely moved on from container-specific security issues, such as the fact that many Docker images retain root access to the host by default, which goes against security best practices. However, this is often because third-party vendors still package application software and assume container images have legitimate root access to a server, Young said. Alternatives that disallow root access by default, such as SELinux containers and user namespaces, must still work out significant kinks in development to be production-ready and manageable to enterprise users.
SELinux containers, in particular, are difficult to set up correctly, because Linux, too, was designed to run on servers, Young said. The containers' setup requires careful crafting of the container file, with many discretionary choices for DevOps pros that may not be well-versed in security.
User namespaces require integration with Linux operating systems and file systems to enforce their permissions, and such integrations have yet to become generally available, said Dan Walsh, a distinguished engineer at Red Hat who also spoke at the panel.
Korren said he's concerned that IT pros who are not well-versed in security will see news such as etcd encryption and think container Kubernetes security issues were solved upstream.
"When they make big announcements of things [such as etcd encryption], it's causing the uninformed to think that now their workload is secure," Korren said.
Security-savvy shops deploy defense in depth
It's better to address Kubernetes security issues with a combination of third-party tools and technologies, according to early enterprise adopters.
Starbucks Corp. deploys an interlocking chain of tools to protect its Kubernetes clusters from exposure to the open internet, such as an ingress controller from NGINX, a web application firewall from Signal Sciences and a homegrown tool called the Ingress Orchestrator. These tools mean Starbucks platform engineers don't have to manually provision namespaces each time developers launch a service.
Aaron Jahoda of Signal Sciences and Ryan Hild of Starbucks Corp. present on Kubernetes security.
But don't try this at home, said Ryan Hild, DevSecOps engineer for Starbucks. The company began this project back when few tools were available.
"As time goes on, there will be more tools," Hild said. "Don't build this yourself if you don't have to."
Smaller Kubernetes deployments rely on third-party security monitoring and enforcement tools from vendors such as Aqua, Sysdig and Twistlock, and they use an immutable infrastructure approach to limit unauthorized activity among application containers.
"Making sure services that shouldn't have access to others don't talk is easy to provision for VMs with security groups on AWS. But in a container cluster, it's more difficult," said Travis Jeppson, director of engineering at Nav Inc., a small-business credit advisory firm based in San Mateo, Calif. "The Twistlock Defender monitors our hosts and containers, and if files are changed, Twistlock turns the container off and Kubernetes spins up a new one, so our environment is continually in a state we know [is secure]."
Twistlock container image scanning also compares container images as they're promoted through a CI/CD pipeline at Nav, and it compares deployed images against those kept in the Docker registry to ensure they are secure and verify their provenance before they're deployed in production.
Container security requires innovation and risk
Making sure services that shouldn't have access to others don't talk is easy to provision for VMs with security groups on AWS. But in a container cluster, it's more difficult.
Travis Jeppsondirector of engineering, Nav Inc.
There's always more to do in container security, given the many layers of abstraction involved with Kubernetes. In many cases, this work requires cutting-edge approaches to IT security.
Some of these IT architectural approaches come with a steep learning curve. Nav will evaluate Istio and Linkerd service mesh to enforce traffic rules between microservices, but the firm will check out vendors at KubeCon for ways to make service mesh deployment and management easier.
"Service mesh changes a lot, and it's really not a drop-in solution yet," Jeppson said. "It's going to take time to support it and get it to work well."
For now, Twistlock enforces traffic rules between microservices, but a service mesh would add another layer of protection, Jeppson said. It would block traffic between services that shouldn't communicate, while Twistlock would verify that enforcement.
"The saying in security is, 'If you have one [tool], you have none,'" Jeppson said.
Some Kubernetes security issues involve risks beyond the technology itself. Vendors on the forefront of Kubernetes security are often fledgling startups without the stability of established security software makers, but established IT security vendors must still catch up with the latest container security issues.
Nav was caught in this Catch-22 when its ingress controller vendor, Turbine Labs, went out of business just as the company deployed Kubernetes in production in July 2017. Nav switched to Traefik.io's open source ingress controller, but will evaluate several others for long-term use, such as NGINX, HAProxy, and Ambassador, Jeppson said.
