Google Chrome experiment crashes in RDS. What can we learn?
Environments using Google Chrome experienced some panic last week as tabs went blank.
Last week, enterprise admins and office workers were less than excited to discover that Google Chrome kept crashing and leaving users with empty, white tabs in RDS and Citrix environments. In tightly controlled environments, this incident was not a confidence booster for using just Chrome.
Let’s look at what happened and whether there’s anything admins can do going forward to prevent this (if it’s preventable).
A Google experiment goes live
Google decided to enable an experimental flag called WebContents Occlusion to stable Chrome versions (77/78)—spoiler: it didn’t go very well. The feature is designed to reduce resource usage when not in active use (which given how resource-hungry Chrome is, is a good idea), it just didn’t work as intended. Instead, Chrome would unload the tab entirely when it sensed another app in active use.
The feature had been in beta for about five months, according to the Chromium bug thread. It was already live on about 1% of users’ devices and as Google didn't notice any complaints or issues, they went ahead made it live on the stable version of Chrome. (It’s not clear why Google didn’t do a staged rollout.) One comment in the thread mentioned having issues with Chrome before last week and wondered whether they were part of the 1%--it’s not clear.
Unfortunately, the experimental feature didn’t work well for virtual environments and left workers staring at blank tabs if they used RDS or Citrix (we didn’t happen to see any mentions of this happening in VMware Horizon environments). While some users might be able to jump to another browser (provided the web application they use is multi-browser), this isn’t always an option for tightly controlled workplaces.
Can you prevent this in the future?
So, it turns out Google can just decide to subject every user to new features without them even knowing. Is this something new admins just have to learn about and be worried about or is there a way to avoid it? There’s a little good news, but mostly bad news. First, the bad news.
It doesn’t matter how you manage Google Chrome—none of the techniques would have prevented the experimental flag from going live. Since Google didn’t change the version number of Chrome when adding the feature (which, once stable, seems like a nice improvement), disabling auto updates and the Group Policy “pin Chrome Browser to a specific version” wouldn’t prevent this from happening again. Disabling component updates might be one option, but it should be noted that some components are exempt, so it’s not clear how helpful this would be overall. You could stay a couple more versions behind (this didn’t affect Chrome 76), but then you open yourself to other potential problems.
The good(-ish) news is that there are ways to mitigate this particular flag and others in the future—but only after the change has gone live. While Google has since rolled back the change, admins could go into chrome://flags and disable #web-contents-occlusion and #calculate-native-win-occlusion. Some commenters in the Chromium bug thread found success with the Chromium command line --disable-backgrounding-occluded-windows.
Admins can also disable experiments with the --no-experiments Chromium command line, but it doesn’t disable about:flags, so admins would have to start up Chrome with the command line and then disable that particular flag. One commenter in the thread did mention that they have a limited command line length in their RDS/Citrix environment, making this difficult to use.
Chrome isn’t the only browser with the ability to rollout features without altering the version number, Firefox does as well, though it can be outright disabled. Additionally, all Chromium-based browsers have this "feature," meaning it'd be in Chromium Edge.
Final thoughts
While Google apologized for the crashing, they didn’t exactly provide much in the way of reassurance that they wouldn’t do this again in the future. Google Engineer David Bienvenu said, “I hear you loud and clear bout (sic) being able to opt out of these experiments. I don’t know if it’s currently possible, but if not, I’m sure it will be discussed.”
While it’s possible to disable the flag through chrome://flags, it’s not currently possible to disable any experimental flags from becoming enabled across an organization, only fix the issue afterwards.
Maybe Google introduces the ability to prevent experimental flags via group policy and the Google Admin Console going forward? Another easy method to fix this is that maybe Google should actually change the version number anytime they do anything to Chrome that will affect live environments. That alone would have prevented this from affecting organizations that disabled auto updates. Or, maybe there should be a long-term service release of Chrome.
Has this event caused your organization to consider moving to another browser like Chromium Edge (so do you trust Microsoft not to use it) or Firefox?
