Client virtualization, part 1: Is it the past or the future? (With a retrospective on XenClient)

The “two worlds” concept

My initial memory was that the idea behind client virtualization was simple: Run a hypervisor on a laptop, and then run two VMs with separate copies of the OS on top of it. An enterprise could issue a laptop to an employee, and then the employee would work in the trusted VM, where the enterprise had locked down the apps, added all the antivirus software they wanted, and so on. And, if users wanted to check Facebook or personal email, they could enter the personal VM.

This is a type-1 (or bare-metal) hypervisor model; and of course, many people are also familiar with type-2 hypervisors, where a hypervisor and guest OS run as software above the natively installed OS. The debate on hypervisors and their merits raged several years ago, but it’s not much discussed these days, although we have plenty of articles that explain the differences for those curious.

Type-2 hypervisors like VMware Fusion and Parallels Desktop are popular today, mostly in compatibility use cases, e.g., running Linux apps on a Windows PC or running Windows apps on a Mac. However, the products that I am focusing on are the type-1 variety, with the interest driven by security.

First, let’s look at why type-1 gets all the security interest.

Trusted root and rings

An OS usually runs its kernel code in Ring 0. (Protection rings are a computer science concept that describe different levels of access to resources.) Ring 0 is the level with the most privileges and interacts most directly with the physical hardware, such as the CPU and memory.

Above this, you typically find less privileged functionality, such as device drivers in Ring 1 and user applications up in Ring 3. Lower rings are able to block higher ones, and only allow controlled access or a subset of functionality via hardened interfaces.

There are also rings below Ring 0, from -1 (as in negative) to -3 down at the hardware level, which again, can police higher rings. Overall, anything lower in the stack can know what the state of the world should look like, and present the rings above with a limited or contrived version of the world.

A user application (typically in Ring 3) running on a compromised OS isn’t secure (and probably can’t even detect this), because the OS underneath the app can tell it whatever it wants, intercept the application data, copy the screen, key log, and the like.

Therefore, the road to true security is to get in as low as possible in the ring hierarchy, using small hardened and trusted bits of code and very limited interfaces, so you can detect and block anything trying to mess with higher rings. This ensures that code in higher rings can rely on functionality made available to them by lower rings. Something running in Ring 0 may be able to rely on functionality in Ring -1, or even down to hardware (e.g., Intel TXT) to query if the system is known to be compromised.

Today, type-1 hypervisors typically run privileged code at lower levels, as well. The hypervisor is run in Ring -1, and things like UEFI plug-ins run in Ring -2. You can essentially have a mini-hypervisor in Ring -2, providing a level of security services, while the full hypervisor in Ring -1 deals with all the driver voodoo and complexity. A mini-hypervisor at Ring -2 can watch for anything related to VMEXITs and can mess with EPT and shadow/virtual EPT to implement cut/paste-like restrictions.

On boot, Rings -3 & -2 boot, and if the OS supports a hypervisor at -1, then it boots, too. Since modern Windows embeds Hyper-V, if the switch for Hyper-V is installed, Windows boots into Hyper-V mode, which gets Windows into VM mode (or another hypervisor can be used). As long as a product gets in early enough, micro-hypervisor elements can be popped in, and nested hypervisors are perfectly feasible.

Type-1 hypervisors include Xen, XenServer, Hyper-V, VMware ESXi, etc., and for security-focused companies, this is definitely the way to go when looking to secure a trusted root as low down the stack as possible. (This was the route taken by the original XenClient XT product.)

Type-2 hypervisors run on top of an OS. An OS has a large footprint to secure, so shoving one between the hardware and the hypervisor makes securing the hypervisor much harder.

The rise and fall of XenClient: The genesis

Now that you’ve had a thorough lesson on hypervisors, let’s look at Citrix XenClient.

I spoke to Ian Pratt, who led the development of XenClient. Ian was a founder of the open-source Xen hypervisor project, which now is included in major virtualization platforms like Citrix XenServer, Amazon AWS, and Huawei’s hypervisor. He joined Citrix when they acquired XenSource, and he’s now the CEO and co-founder of Bromium, which I’ll talk about in the next article.

Talking to Ian, it became clear that while the XenClient prototypes developed around 2008/2009 had been driven by interest from security and intelligence agencies, a large internal driver was the potential for a solution that could simplify management and patching for the ever-growing number of laptops, mobile devices, and workstations, at a time when patching Windows wasn’t easy.

The original XenClient product became known as XenClient XT, and was released in 2010. XenClient XT was a “microvisor” (micro hypervisor) based around the Xen type-1 hypervisor. Like the original model I described, you would install on a laptop, and the user would run two VMs: one containing their “personal laptop” (the untrusted world) and one containing their “corporate laptop” (the trusted world). This approach was attractive to very high-end security customers, financial corporations, and federal security use cases.

It all got very confusing in 2012 when Citrix acquired the NXTop product by way of their Virtual Computer acquisition. NXTop eventually got rebadged and evolved into XenClient Enterprise.

XenClient: The results

Security-wise, all those involved seemed to agree that XenClient XT was disappointing. Whilst the model worked as per the theory, the corporate secure VMs were only as good as their installed security. In other words, they were just like any other corporate laptop. The VMs used were persistent and, once compromised, issues persisted beyond a single login. Users doing perfectly legitimate tasks continued to pick up viruses and other nasties, and managing all the antivirus and other security tools was as cumbersome as ever.

Meanwhile, a lot of the issues with the management of laptops went away: the Microsoft patching process got better, the patches themselves got more reliable, and a clutch of really good tools around testing/deployment appeared (e.g., SCCM, BigFix, Alterian, and Rapid 7).

On the other hand, using products like XenClient meant adding more infrastructure and support staff that was just for laptops.

Hardware compatibility was a huge issue for all the XenClient products. Dan McCall, who was the CEO of Virtual Computer, gave me insight into the scale of the challenge:  

“It is simply the case that all the different variations of hardware required a tremendous investment in QA and engineering. Unlike Microsoft, who rely on PC vendors to do their hardware QA, Virtual Computer Inc. (our small 35 person company) had to do QA on ALL the laptop PCs our customers wanted to support. This forced us to create a Hardware Compatibility List (HCL) that covered barely 10% of the market. The older the hardware, the worse the problem (we were buying old PC’s on EBay to do QA with!). Eventually we secured Lenovo and a relationship with an innovative senior executive, able to help us w/with the QA issues as well as with Lenovo’s hardware suppliers by asking them to provide us with Linux drivers for their new hardware (think of how many different versions of something as simple as a touch-pad there are). That kind of market influence can’t be obtained by a startup!”

Plus, there was another huge issue here: asking the user to choose a trusted/untrusted world and to make an assessment on whether something is a dubious concept at best.

You know where this went. The uptake was low, and Citrix axed XenClient in 2015. XenClient XT was open sourced as OpenXT, presumably because of some continued demand or contractual obligations to customers. (If Citrix open sourced every product they’ve culled, it would be expensive—so this was unusual.) The OpenXT repositories continue to be updated and actively developed; open-source software, particularly in security, is popular for many federal agencies, so I imagine it may be these sorts of organizations that are using and developing it.

Citrix stopped selling XenClient Enterprise, turning to focus on type-2 hypervisors with DesktopPlayer. DesktopPlayer for Mac had started as a product to give Citrix some sort of coverage for Mac endpoints with XenClient-like use cases. When XenClient Enterprise’s demise was announced, DesktopPlayer was still Mac only, although we covered rumors of a Windows version, which eventually appeared. Last year, DesktopPlayer seems to have silently gone to EOS (End of Sale) on August 31, 2018, with an EOL (End of life) date of September 28, 2019 (see the Citrix product support matrix for details).

It was a different time

Dan provided some great insight into the sentiment of shareholders and venture capitalists back in 2010 to 2012. Basically, Windows PCs fell out of fashion with the advent of the iPhone and iPad. Today, with a billion PCs deployed and all needing to be refreshed, it seems hard to imagine, but at the time, PCs and laptops felt like the past and a saturated market. Due to the technology challenges, this was going to be a very long journey and one that Citrix opted out of.

Both Ian and Dan also mooted that cloud and VDI were still quite niche at the time and security and data breaches just weren’t high enough enterprises priorities. Often, customer interest in XenClient was for small numbers of seats for high-value users. Back in 2010, there weren’t relentless security breach headlines, and corporate regulation and governance (i.e., who is legally on the hook for a breach) weren’t high on the agenda  for most organizations.

David Gaunt, who is now at Nutanix but was a Citrix SE from 2007 to 2014, confirmed this:

“I never understood the need for it and neither did the customers I spoke to, as an SE at Citrix. Very niche use case and while the idea was sound, it was a dead end. It seemed like an idea looking for a market that was never there. Per app and device management took over (MDM, etc.) and that killed it from what I saw. No doubt microvisors have their market but it’s not mainstream enough for a company of Citrix’s size to run with it.”

Wrap up

This article may be ending on a negative note, but that’s only half the story. Next week we’ll return with the resurgence of client virtualization.