Kubernetes on Windows matures, but networking snags remain

Kubernetes for Windows on-premises still needs work

IT pros that want the full advantages of Kubernetes containerization -- such as broad support from cloud provider services and third-party IT management tools -- on familiar Windows Server infrastructure have had a long wait.

Kubernetes on Windows development began in 2016 with Kubernetes 1.5, but it didn't reach GA in the main code line until Kubernetes 1.14 in early 2019. Microsoft's Azure Kubernetes Service has offered GA support for Windows Server Containers since only late April, and Docker Enterprise just began to support production-ready Windows nodes with version 3.1 in late May.

Windows-only shops that manage their own on-premises infrastructure still struggle the most to reach production with Windows on Kubernetes. Some may argue that is a shrinking audience, as the shift to Linux and Azure is Microsoft's clear priority for the future, and many enterprises have fully embraced public cloud. But it's still a problem, especially in highly regulated industries.

Johan Spännare

"Right now teams at SEB use an Agile way of working, which means if they develop the stuff, they run the stuff," said Johan Spännare, a senior developer at SEB, a Swedish financial services company that runs primarily Windows applications in an on-premises infrastructure due to compliance requirements.

SEB has apps that run in .NET Core, but that doesn't make Linux an option for all of its developers, Spännare said.

"We have teams that are trying to do things in Linux, but the majority of our developers will stay on Windows," he said. "We can't use Kubernetes until we have [stable] Windows nodes available."

That's not to say there hasn't been progress. This year's Kubernetes releases, 1.18 in April and 1.19 in late August, continued to bring Windows node support closer to parity with Linux. Version 1.18 added an alpha version of a Container Storage Interface (CSI) proxy meant to stabilize Windows hosts' access to external storage systems, which moved to beta in 1.19.

Version 1.18 also introduced experimental support for the open-source containerd runtime, in beta as of 1.19, which will give Microsoft and Kubernetes developments a common component with which to work . Previously, Microsoft supported only the full Docker runtime, which has a larger footprint. It also lacks features such as strong pod isolation within clusters for security and the ability to mount single files, rather than folders, into containers from external filesystems. Containerd support was made possible by new features in Windows Server 2019, the only supported OS version for Kubernetes nodes.

Kubernetes on Windows networking issues persist

While Windows Server 2019 was a major release, succeeding Windows Server 2016, it doesn't have everything needed to run Kubernetes smoothly.

Microsoft did a lot of work redesigning the whole networking stack [in Windows Server 2019], and it's much better, but we still see some issues.

"Microsoft did a lot of work redesigning the whole networking stack [in Windows Server 2019], and it's much better, but we still see some issues," said Daniel Terry, a solution engineer at SEB.

Windows nodes don't support iptables, a key Linux OS construct that helps Kubernetes clusters efficiently orchestrate network operations at scale. This and other Windows networking issues that are still being worked out upstream also present headaches for IT pros.

For example, Windows uses the Windows Host Network Service (HNS) to create network resources such as vSwitches and NATs. Kubernetes creates an HNS load balancer for every service in a Windows cluster, which can lead to network congestion and slow performance, especially during initial application deployments. There are also known issues with using Windows virtual IP addresses (VIP) at scale in Kubernetes.

Daniel Terry

"We've been fighting a lot with Microsoft," Terry said. "The networking stack inside of Windows has been troublesome for us, and a lot of their response has been, 'run it on Linux' or 'move to Azure.'"

Some of these problems could be resolved in Windows Server 2020, first released in May. Many enterprises, including SEB, will still run Windows Server 2019 for the foreseeable future, however.

The slow development of Kubernetes on Windows means some developers now face the same learning curves and cultural hurdles that Linux developers did when Kubernetes first emerged.

"Windows [developers] without experience with containers are still struggling with some of the concepts, things like, 'how do I patch this?'" said Phil Fenstermacher, a systems engineer at William & Mary, a university in Williamsburg, Va. "They're looking for the same wins with containers we've seen in Linux, like reducing server sprawl, but they're trying to assimilate more than just tech -- it took six or seven years for the team I'm on to completely change our culture and do what we're doing now."

Thus, for Fenstermacher, the development of the CSI proxy is promising, but not yet applicable to his organization's Windows environments.

"The applications we're trying to run in Windows containers were selected because they're simple and don't require persistent storage," he said. "Once we're more mature in our use of Windows containers, we'll almost certainly be taking advantage of the CSI Proxy."

SEB has more immediate needs for stable storage interfaces for Kubernetes on Windows. For Terry, however, the adoption of tools such as CSI-proxy may be slowed by their own slow growth, which would delay vetting by enough users to ensure enterprise-level stability, and create a vicious cycle.

"I don't think that many companies have actually implemented it," Terry said. "To be honest, 80% of the workloads running in containers are Linux."