Kubernetes is now in mainstream enterprise use but still must support other audiences, from cloud service providers to high-performance computing, or HPC, shops. Tension among these varied interests surfaced this week in discussions about whether the upstream project should include more security defaults.
The platform has developed into a stack of components, including separate storage and networking interfaces and layers of deployment templates. This factor, coupled with a trend toward multicluster and multi-cloud management, threatens to overwhelm IT operators with Kubernetes management complexity.
Kubernetes security configuration is also a multifaceted process that encompasses container-level, host-level, Pod-level and cluster-wide policy settings.
This approach is flexible for customization of upstream Kubernetes by cloud service providers and IT vendors, such as Red Hat and VMware. It also avoids conflict between default settings and performance requirements in specialized environments, such as HPC. But, for general use, Pod Security Policies are especially a headache, according to IT pros.
"They have very fine-grained control ... but they're applied at a cluster level, which is quite broad," said Shane Lawrence, senior infrastructure engineer in cloud security at online retail service provider Shopify. "You can use service accounts with specific RBAC [role-based access control] permissions in a given workload's namespace to ensure that the most appropriate policy for that workload is applied, but it adds a lot of complexity."
Shane Lawrence
There are ways to mitigate this from a security perspective, Lawrence said, namely by deploying separate tools, such as Falco, an open source security policy tool Shopify uses to ensure that Pods aren't granted inappropriate privileges.
"It still feels more complicated than it needs to be," Lawrence said.
Kubernetes Pod Security Policies in limbo upstream
Pod Security Policies are a prime example of the current impasse in Kubernetes security. The subproject has been stuck in beta since Kubernetes 1.10, released in 2018. It has been in beta so long that it inspired a new rule, as of the Kubernetes 1.19 release slated for Aug. 25, that limits the number of releases an API can stay in beta before it must be made generally available or deprecated. Kubernetes 1.19 also contains three new Pod Security Standards that package basic settings for privileged, baseline and restricted environments to simplify their operation.
Kubernetes maintainers are aware of users' plight with Pod Security Policies but haven't yet reached consensus on how to proceed. There is some precedent for expanding core Kubernetes to include security defaults, such as RBAC, made generally available in version 1.8.
A native extension for Kubernetes multi-tenant security called hierarchical namespaces was also added in Kubernetes 1.19. But, generally, keeping the core project sharply focused has been a founding principle for the Kubernetes community. This principle has helped Kubernetes avoid the stagnation and overcomplication faced by other open source infrastructure projects, such as OpenStack.
Upstream Kubernetes maintainers must also respect the territory of vendor-specific Kubernetes distros that offer Kubernetes security by default, including Red Hat OpenShift, which donated its admission controller as the basis for Pod Security Policies upstream.
"This is very similar to what happened with Linux," said Kirsten Newcomer, director of DevSecOps strategy at Red Hat, in a panel session at KubeCon Europe Virtual this week. "Linux has a lot of complexity as well, and you look to distributions to provide security out of the box."
Since Pod Security Policies were introduced, open source tools, such as Open Policy Agent (OPA) and its Kubernetes-specific Gatekeeper admission controller, have also emerged, leaving the community with a dilemma about whether to continue developing similar features as part of the core project.
"Even though CIS [Center for Internet Security] benchmarks recommend Pod Security Policies, the community's still not convinced about that, and I'm starting to see a drift toward OPA and Gatekeeper," Newcomer said.
Enterprises want help with Kubernetes network policies
Enterprise users are aware of the risks associated with adding or changing upstream Kubernetes security defaults.
Jennifer Strejevitch
"It may put someone new to Kubernetes off to test the product in a development environment and find out that they need to get Kubernetes certified ... before they can see the full feature set, if everything was secured by default," said Jennifer Strejevitch, site reliability engineer at publisher Condé Nast. "It would be good to discuss the downsides of more restrictive network policies by default ... [and] the risks of changing those defaults in newer versions."
Panelists in a KubeCon HPC session added that secure-by-default Kubernetes distributions may restrict observability tools' access to clusters and introduce performance overhead.
"Having easier-to-access security is absolutely a good idea and not one I want to discourage," said Marlow Weston, lead HPC tools engineer at Intel, in a Slack Q&A discussion that followed the panel session this week. "[But] it depends on what the defaults are. It would be better to put in an opt-in model and let the people making their systems evaluate what the actual performance impacts are and then move [on] from there."
It's really easy to edit a network policy and make your Pod accessible from the entire cluster -- that's the default.
Jennifer StrejevitchSite reliability engineer, Condé Nast
However, there's also risk in complex Kubernetes security configuration, especially for developers that aren't experienced with the container orchestrator and concepts such as network policies, Strejevitch said.
"It's really easy to edit a network policy and make your Pod accessible from the entire cluster -- that's the default," Strejevitch said. Even if a Pod has RBAC configured, without network policies that restrict communication among Kubernetes namespaces, users can still potentially access other Pods.
"We keep applications from different projects in different namespaces," she said. "[But] having some sort of default 'wall' between Pods that we can rely upon would be less time-consuming."
From an IT security-focused perspective, secure Kubernetes defaults would be reassuring, said Jay Beale, principal security consultant at InGuardians Inc., an independent IT security consulting firm in Washington, D.C., and co-leader of the Kubernetes SIG-Audit working group.
"I do penetration and security architecture review against companies' Kubernetes clusters at InGuardians, and we see a lot of clients not have any kind of Pod Security Policy or admission control at all -- it's 'something they're going to get around to,'" Beale said. "We have to make this easier for people."
Arguments against secure upstream Kubernetes defaults are understandable, Beale said, but for enterprises, the struggle with complexity is a real issue the community must address.
"I see the reasons on both sides," he said. "If there is one thing to use by default but you can use another tool if you want to, it lets us have some ease in the conversation around convincing operators to use it."
Shane Lawrence
Jennifer Strejevitch
