Enterprises that pay for Kubernetes support will respond differently to this week's critical security vulnerability than lean DevOps teams that use upstream distributions.
A critical Kubernetes security vulnerability that was patched this week affects a wide swath of container orchestration users.
CVE-2018-1002105 is a flaw that enables attackers to escalate privileges via the widely used Kubernetes API server. With Kubernetes reigning as the de facto standard for container orchestration in enterprise IT, the flaw poses a potentially huge risk for the IT industry as a whole.
The Kubernetes open source community issued a patch upstream for version 1.10 and above, and it was already applied to hosted Kubernetes services, such as Google Kubernetes Engine and Red Hat OpenShift Online. However, users with on-premises deployments or self-hosted Kubernetes clusters in the cloud will have to roll out the patch themselves.
"There have been other Kubernetes flaws in the past, but this is definitely the worst," said Gary Chen, analyst at IDC. "This is always the worry with centralized control planes -- if someone hacks it, they can get access to everything else."
Darren Shepherd, co-founder and architect at Rancher, based in Cupertino, Calif., reported the vulnerability to the Kubernetes security team last week. The flaw was patched this week, and the flaw and patch were simultaneously disclosed after a weeklong embargo. Kubernetes 1.10, which is the earliest version of Kubernetes supported by the open source community, came out in late March 2018. While customers who use the API aggregation layer that became generally available with Kubernetes 1.10 may have an increased attack surface, the vulnerability may affect Kubernetes versions going back to version 1.7, when a beta version of the aggregation layer was first enabled by default.* Still, version 1.10 has only been in the wild for about 9 months, and came out in late March 2018, so the vulnerability has been in the wild for about nine months. Kubernetes contributors, such as Red Hat, said that's a short window compared with other critical open source vulnerabilities, like Heartbleed, Spectre and Meltdown, which were exploitable for years.
"It's always terrible to find a severe problem, but the Kubernetes team took this very seriously and issued a fast response," said Chris Robinson, product security assurance manager at Red Hat, based in Raleigh, N.C. "That's a trend we've seen with CVE embargoes in open source, in general, the last five years. On average, they've gone from 14 days down to about 10."
Tech vulnerability exposes market disparities
However, some IT pros said only large IT teams with packaged Kubernetes implementations can keep up with quarterly updates to the orchestration tool, and this Kubernetes security vulnerability presents an additional roadblock to timely upgrades past version 1.8.
Editor’s note:A previous version of this article reported that the CVE only affected Kubernetes versions 1.10 and higher, which is not the case. Version 1.8 of Kubernetes is also potentially affected by the bug, said Rancher’s Darren Shepherd, in an interview that took place after this article was published, though he said the attack surface for the CVE was potentially bigger in versions where the API aggregation layer is used, and in multitenant environments.
"Kubernetes may be the standard overall, but within it, there are multiple standards to pick from between versions 1.8 and 1.12," said Rick Moss, infrastructure operations engineer at MailChannels, an email service provider in Vancouver, B.C. "There are many companies facing the 1.10-plus environment, with its requirements for TLS [Transport Layer Security] and namespace controls, slowly because the [new requirements] will make development and deployment cycles more complex."
Worse, many enterprises freeze production updates and deployments between early December and mid-January, making the requirement to patch this Kubernetes security vulnerability particularly ill-timed in Moss' view.
"If you want to ruin Christmas, a major CVE in early December is pretty much how you do it," Moss said. "But, if you need to deploy this patch, not doing so could really change your Christmas plans, too."
Companies with big IT teams dedicated to infrastructure support who have the money to buy commercially supported Kubernetes packages will have an easier time than lean DevOps teams that may not be well-versed in infrastructure security. This outcome could well amount to an argument in favor of packaged distributions over upstream code, but the reality is that cost is king in the open source software world, Moss said. He predicted that companies without the resources to respond to this kind of Kubernetes security vulnerability will instead delay upgrades or simply ignore the problem.
It's not that we haven't seen vulnerabilities like this before, but this is a new problem based on the religion of DevOps and small teams.
Rick Mossinfrastructure operations engineer at MailChannels
"It's not that we haven't seen vulnerabilities like this before, but this is a new problem based on the religion of DevOps and small teams," Moss said. "The truth is that large-scale infrastructure deployments need a dedicated team to monitor the back end and keep it up to date, and smaller teams can get stuck."
Some upstream adherents said they won't have a problem applying the patch, but the possibility of future Kubernetes security vulnerabilities affected their operating model with the open source distribution.
"We have application-specific clusters, and those are only accessible by the users that have full admin [access] to those clusters," said Adam Brown, co-founder of Mux Inc., a video streaming startup in San Francisco that serves media giants such as CBS and PBS. "We expect this to change in the future, and bugs like this are pretty terrifying. But, at the moment, it doesn't appear to be exploitable in our clusters."
IT vendors rush to patch Kubernetes security flaw
IDC's Chen predicted this Kubernetes security vulnerability and others that could arise in the future will push enterprises to invest in application security tools and take another look at packaged Kubernetes distributions from industry vendors.
"This is going to be something enterprises look at, in terms of vendors: Which ones responded the fastest and provided the best information?" Chen said.
As scary as this vulnerability was, the fact that Kubernetes' maintainers found and quickly patched it is a point in favor of Kubernetes maturity overall, Chen added.
In the wake of the vulnerability disclosure, Red Hat, Mesosphere and Google issued public posts that notified users of the patch. AWS, Microsoft Azure, Docker and Rancher did not respond to inquiries about whether they'd applied the patch to their products as of press time.
