Kubernetes security project faces reckoning over beta status
Kubernetes Pod Security Policies could be marked for deprecation as soon as the next Kubernetes release, in the wake of new limits on the beta phase for components of the platform.
Spurred by new rules against lengthy beta stages, Kubernetes security contributors have agreed to mark Kubernetes Pod Security Policies as deprecated and begin work on a replacement.
The Pod Security Policy API, which limits what Kubernetes security features pods and containers can use, was first introduced in Kubernetes version 1.3 in 2016. It has remained in some form of beta since then, so long that it inspired a new rule -- first discussed as of Kubernetes 1.19 a few months ago -- that projects cannot stay in beta for more than two Kubernetes version releases before they must be moved to stable status or marked for deprecation and eventual removal from the platform.
Despite its beta status, the Pod Security Policy API is used by enterprises in production, and by cloud providers such as Amazon EKS. But the API has been plagued with issues that seem unlikely to be resolved at this point, according to a document by Tim Allclair, a software engineer at Apple and a co-chair of the Kubernetes Special Interest Group SIG-Auth, which oversees Pod Security Policies.
Pod Security Policies in their current form must be bound either to a human user or a Kubernetes service account, the document states. This results in a dual policy enforcement mechanism that's confusing to implement and potentially weakens security.
The API "fails closed" in the absence of policy, completely restricting access and potentially disrupting applications, but policies can't be enabled by default. The community has struggled to maintain a consistent scope for the API as it has developed, and doesn't work well with service mesh sidecars or CI/CD validation tools, making it difficult for users to test policies.
The issues raised with Pod Security Policies are ones that we see preventing it in its current form from moving forward.
Tim AllclairCo-chair, Kubernetes SIG-Auth
"The issues raised with Pod Security Policies are ones that we see preventing it in its current form from moving forward," Allclair said during a presentation to open the Nov. 11 SIG-Auth meeting.
During this meeting and a follow-up on Dec. 9, SIG-Auth and SIG-Security reps decided to mark Pod Security Policies as deprecated and come up with a new project to completely replace them.
Enterprise IT pros aren't exactly mournful to see the API shelved, although many have already put in hard work to make Pod Security Policies function well in their environments.
"They are a bit obtuse and difficult to use," said Matt Young, principal cloud architect at online insurance marketplace provider EverQuote in Cambridge, Mass., who has had to wrestle with configurations for observability tools such as Loki log monitoring and Grafana Labs' Cortex data store, given his company's stringent pod security policies. "Many things that just work in demos make the assumption you have a highly privileged default Kubernetes service account."
Young has used a third-party tool, Sysdig's Kube PodSecurityPolicy Advisor, to scan Kubernetes configuration files in a test cluster and create pod security policies before deployments to production. But this still isn't an ideal solution, he said.
"PSP Advisor is a nice way to start, and it enforces some naming conventions," Young said. "If you're doing a bunch of these, it's awful -- I'm glad [the community] is going to make [Pod Security Policies] better."
Kubernetes SIG-Auth co-chair Tim Allclair gives a presentation on the future of Pod Security Policies at the group's Nov. 11 meeting.
Kubernetes security groups start over with pod policies
Kubernetes security maintainers have begun work on a wholesale replacement for the Pod Security Policy API that won't maintain compatibility with the original, a decision first hammered out during the Nov. 11 SIG-Auth meeting and affirmed among members this week.
"The API as it is today is not workable, and because … APIs in Kubernetes remain compatible across version boundaries, I don't see semantic behavior change as an evolution that can happen," said Mo Khan, an OpenShift software engineer at Red Hat, during the Nov. 11 meeting. "I would personally welcome a KEP [Kubernetes Enhancement Proposal] for an alpha API that starts over and does it right."
SIG-Security co-chair Tabitha Sable was appointed at the November meeting to begin drafting a proposal for Pod Security Policies' replacement. At the Dec. 9 meeting, Sable said she was still working on the document, and would circulate it as a Google Doc for input from the community before formalizing a KEP.
"There's work ongoing before it's something specific enough to agree or disagree with," Sable said this week.
Maintaining compatibility with the original Pod Security Policy API would be ideal from a Kubernetes upgrade standpoint, maintainers agreed, but it would also likely mean any replacement API would suffer from the same problems as the original.
"The shape of the existing API and maintaining compatibility with a thing that has a ton of problems actually made it harder to get to a good place" in previous work to improve the project, said Jordan Liggitt, a Google software engineer and SIG-Auth technical lead, during the Nov. 11 meeting. "Rebuilding the airplane in flight, I think, will just waste a lot of energy keeping compatibility with the bad parts of PSP that we wish we could drop."
Pod security policy deprecation timing not yet formalized
Given the new beta rule in version 1.19, SIG-Auth could take no action, and the API will be marked for deprecation automatically as of Kubernetes 1.22, likely to ship in June 2021 per the customary Kubernetes quarterly release schedule. It would then be removed from Kubernetes entirely as of version 1.25 in early 2022.
However, members of Kubernetes SIG-Auth and SIG-Security discussed marking Pod Security Policies for deprecation as soon as the next Kubernetes release, version 1.21, which is expected in March. Doing so wouldn't change the plan to remove them as of version 1.25 and would give users and community members more time to focus on a replacement.
"Given that we don't anticipate graduating the current API, I think it's useful to mark it as deprecated, just as a signal that if you're not already using it, don't start using it," Allclair said at this week's meeting. "Don't jump on this bandwagon -- wait for the thing we're working on if you're new to Kubernetes."
The group didn't formally decide to mark Pod Security Policies as deprecated in Kubernetes 1.21 during that discussion. SIG-Security's Sable said she agreed that plan would still allow time for public discussion about a replacement API but shouldn't be firmly set at this stage.
"I would want to reserve the right to extend out the end date," she said. "Because what I'd really like to see is for there to be no 'dead zone' where PSP has been removed, but there isn't a good alternative for people to migrate on to -- especially since at this stage of the discussion, one of the goals is to make migrating from PSP as painless as possible for people. I would be opposed to [messaging that says], 'Everybody needs to panic and rush out of PSP now.'"
