What are the challenges of designing virtual reality experiences?

There are technical, physical and experiential challenges to designing virtual reality, augmented reality or mixed reality experiences.

The technical challenges are easier in that we know what a right answer looks like. The devices need to be smaller, lighter, wireless and higher resolution, as well as have a wider field of view, improved tracking, lower latency and so forth. That's all engineering. And, while it isn't easy, at least we know what the targets are.

Physical challenges center around two main areas: exterior and interior comfort. Exterior comfort plays on some of the technical challenges: The gear needs to be lighter, not have wires and be comfortable enough for long-term use -- i.e., hours. At some point in the future, people will likely work in some form of augmented reality (AR) or virtual reality (VR), so eight-hours-a-day use needs to be viable.

Interior comfort refers to issues around motion sickness, cognitive and emotional overload, and possible long-term side effects. There is a rough estimate that about 10% of the population is highly susceptible to motion sickness -- I'm one of them. There are technical parameters that can ameliorate some of the effects: Tracking needs to be fast and accurate; frame rates need to be high -- no or low latency; and the experience needs to be designed in such a way that it avoids certain interactions that can exacerbate motion sickness.

This leads to the longest tent pole in the room: experiential challenges of designing virtual reality, augmented reality and mixed reality (MR) experiences.

We are exceptionally good at telling stories on screens, as we're building on centuries of experience, dating back to the Greeks first creating the proscenium arch, which effectively put content within a boundary. Paintings, movies, television -- all are forms of narrative bounded by a screen border. VR, AR and MR remove those boundaries, and with it, a lot of our narrative forms lose efficacy. Film directors work with cinematographers to determine the best angle to view a certain scene. VR puts that choice into the hands of end users -- if they have that agency -- thus limiting a number of tools and techniques that storytellers have used for decades, if not centuries.

Therefore, new forms of narrative need to be created to design virtual reality, augmented reality and mixed reality to effectively tell stories, as stories are at the heart of how we learn, communicate and are entertained. Right now, we're doing what we've always done when a new medium for communication comes along: We take old content models and port them into the new medium. An analogy is when TV arrived. Some of the first content was taking radio plays and putting them in front of a camera. The world then realized that some people had a "face made for radio." With significant amounts of experimentation, the new mediums are better understood, new models and techniques are discovered and exploited, and the medium matures.

Right now, VR, AR and MR are in the early stages of being a platform for narrative, and most examples are not compelling. This isn't just taking the old stuff and slapping it into a headset. There needs to be a new generation of approaches and techniques of designing for virtual, augmented and mixed reality to enable effective immersive storytelling. In addition, there should be more effective and immersive serious use cases beyond consumer applications, such as training, learning, healthcare, security and so forth.

Have a question for one of our experts? Submit it now. All questions are anonymous.