Meltdown and Spectre patches hit XenApp performance especially hard

What does the testing data show?

Lakeside Software tested the performance of XenApp, Citrix XenDesktop and VMware Horizon after applying Microsoft's patches for the guest and host operating systems and, on virtual servers, a hypervisor patch for VMware ESXi. The tests ran workloads similar to what a task worker would access -- browsers and basic word processing.

For the desktop VM testing with Horizon, Lakeside found a 1.26% increase in CPU usage per VM. These tests ran against only the Meltdown patches, because Spectre patches were not available at the time, the company said. The tests used Windows 10 as the guest OS with a density of 62 VMs per server. Lakeside tested XenDesktop as well but did not publish those findings. Results would be similar across all VDI-based workloads, the company said.

Many are haphazardly patching their systems to protect the vulnerability without looking at the downstream effects.

The XenApp performance hit was much higher. Virtualized XenApp sessions saw a 22% CPU load increase on Windows Server 2008 R2 with an Intel Xeon X5650 processor, and a 19% increase on Windows Server 2012 R2 with a newer Intel Xeon X7350 processor. For physical XenApp servers, sessions with the older OS and processor saw a 20% CPU increase, and those with the newer OS and processor saw a 16% spike.

Lakeside did not test VMware published applications but said it would expect results similar to the XenApp performance. The performance hit occurs because of the way RDSH works -- terminal servers are more dense than VDI servers and have multiple users accessing the same machine -- so it is not specific to XenApp; any workload running on terminal servers will experience the same effects, said Ben Murphy, director of applied engineering at Lakeside.

Chip manufacturers released updates to their microchips' semiconductor material, silicon, using microcode, which Microsoft recommends IT departments deploy in addition to the Windows patches. For its physical server testing, Lakeside did not include those patches because they depend on vendor support, but the company plans to update its testing once it has those hardware updates, Murphy said.

Citrix did not respond to a request for comment regarding advice for customers dealing with performance issues related to the Meltdown and Spectre patches. VMware issued this statement:

"VMware is currently investigating, evaluating and mitigating issues related to the Meltdown/Spectre response. We are updating this Knowledge Base article as new information becomes available. Customers should reference this resource for the latest updates from VMware related to this issue."

For VDI workloads with small CPU increases, an end user would not notice any difference in performance, said Ruben Spruijt, CTO of cloud workspace provider Frame. Citrix shops, on the other hand, are already starting to see noticeable XenApp performance hits from the patches, he said.

"Adding these patches will increase the overhead," he added. "Imagine your performance drops by 20%, and you already maxed out your servers. People will need to buy new gear. That's likely the reality."

Results may vary

The results mesh with what other vendors, consultants and IT pros in the industry said they are seeing. But it's important to note that the specific performance hit will vary depending on a number of factors, including the age and type of server, CPU, hypervisor and guest operating system, as well as the workload.

Login VSI, a virtual workload testing software provider, even saw one customer whose VMs wouldn't restart after the Meltdown and Spectre patches because of a compatibility problem.

"This is a critical patch ... but if you skip your due diligence, you might be in for some trouble," said Mark Plettenberg, senior product manager at Login VSI.

The company is offering its load-testing product for free through the end of March to help organizations test performance after the Meltdown and Spectre patches.

Organizations must do their own testing to determine the consequences of patching their virtual deployments, Spruijt said.

"Benchmark testing doesn't apply one on one to every customer scenario, because different customers will have different applications and different combinations of things," he said. "Try to understand what the impact is for your environment. Then the question is, 'Do you have enough capacity left to handle that impact?'"

The increase in CPU usage happens because of higher throughput on the disk, as a process requires more CPU to handle operations such as video streaming or file transfers that consume more of the disk. So a workload that requires a high disk throughput will see more of a performance hit, Murphy said.

Physical Windows PCs will see some performance degradation as well, depending on the operating system version and processor age, according to Microsoft's own preliminary benchmark testing.

What IT admins can do about it

The first step to mitigating the effects of the Meltdown and Spectre patches is to improve monitoring. IT pros should track two key metrics to determine the performance of VDI, RDSH and published apps: processor time utilization and CPU wait time, Young said.

Processor time utilization indicates CPU utilization across the entire infrastructure, not just peaks and valleys on individual VMs. CPU wait time shows how long a process or application within a VM has to stand by for enough CPU resources to be available on the physical server before it executes a requested task. Both numbers should be low; otherwise, users will see slower performance, Young said.

The Meltdown and Spectre patches can also have consequences for networking and storage. If the processor on the host maxes out, network latency increases and storage performance suffers. Organizations may need to buy additional servers to handle the hit on CPU processing, which would bring additional costs, Young said.

"Many are haphazardly patching their systems to protect the vulnerability without looking at the downstream effects," he said.

It's all in the preparations

Oral Roberts University was in the midst of welcoming 4,000 students for a new semester when the Meltdown and Spectre vulnerabilities became public. The IT team had to quickly test and deploy the Windows patches for its VMware Horizon virtual desktops across campus.

Users have reported no noticeable effects so far, said Michael Mathews, CIO at the university in Tulsa, Okla. In fact, having VDI meant it was a lot easier for IT staff to handle the patching process, because it required patching fewer shared base images, Mathews said.

"We never saw a hiccup," he added. "And if we were not on VDI, they'd be running around to all these PCs and probably wouldn't have gotten to them all."

One managed services provider, which runs Citrix infrastructure in two data centers, started by patching its hosts, then moved onto the individual VMs. The company warned customers about the Meltdown and Spectre patches and potential performance effects, but it hasn't noticed any issues yet.

"We were doing management of customer expectations more than anything," said Michael Thompson, a systems engineer at the provider. "We deal with patching all the time, so that's nothing new. The biggest fallout I see is the questions at the customer level. Everybody is worried."

Thompson has seen a rise in CPU utilization, but there's plenty available because his company overcommits CPU, so users don't feel any effects, he said.

"We try not to under-build anything," he added.