What is mixed reality?

How does mixed reality work?

Mixed reality sits between augmented reality, which overlays digital content onto the real world, and virtual reality, which immerses users in a fully digital environment. Because it combines VR and AR with the physical, mixed reality requires advanced technology to integrate the virtualities seamlessly with reality.

Mixed reality headsets feature a range of hardware to facilitate the integration, including cameras and sensors, as well as a display interface to project different elements, such as a video game scene. Mixed reality also relies heavily on input sensing and tracking movements to process and facilitate real-time interactions between physical objects and virtual elements, a defining feature that separates mixed reality from VR and AR.

Mixed reality technology uses advanced software to process data and information collected by a headset's cameras and sensors. The information collected from the headset is analyzed to incorporate and overlay VR elements accurately into a user's physical environment, which is continuously monitored by the software to update in real-time.

Using data from sensors, mixed reality creates a three-dimensional map of the environment, known as spatial mapping, which provides the geometry of the room and is crucial for accurate interactions between virtual and physical objects.

For example, if the camera identifies a couch in the room, the software uses that information to overlay an NPC onto the couch or place a virtual object on a table. Additionally, the software uses the data to obscure objects behind physical items, creating a more integrated and immersive experience.

The advanced technology also enables users to "anchor" digital objects to physical areas. They remain in place even if a user moves around. For example, a user anchors a digital element, such as a virtual book, to a real coffee table. The virtual object stays there unless moved again by the user.

Applications of mixed reality

Although mixed reality is considered an emerging technology, practical and recreational uses already impact several aspects of society.

Education

Mixed reality facilitates interactive, immersive learning, such as the following:

• Virtual field trips.
• Virtual science experiments.
• Historical reenactments.

Immersive experiences using mixed reality better engage students of all learning styles and offer hands-on participation, leading to more efficient and effective learning.

For remote learners, mixed reality provides a more interactive experience with collaborative learning environments and immersive learning opportunities they wouldn't have otherwise.

Mixed reality is also used in technical training, such as engineering or manufacturing, during which students gain firsthand knowledge without risking safety or costly mistakes.

Entertainment

Entertainment is one of the most common uses for mixed reality and is quickly redefining the world of virtual reality gaming. Using mixed reality, users fully immerse themselves in virtual video games, including interacting with characters and objects, all integrated with their physical environment.

Storytelling becomes more interactive with mixed reality. Using VR elements, users engage in narrative stories and become active participants, making choices to move the narrative forward and exploring the mixed reality environment to engage more deeply with the story.

Healthcare

Healthcare and medical professionals use mixed reality to plan and assist during surgical procedures, including 3D planning and visualization of a patient's anatomy and unique case.

Medical students use mixed reality to practice procedures in a safe, simulated environment. Additionally, students practice other essential skills through mixed reality, such as patient education and surgical preparation.

On the patient side, mixed reality visualizes situations, which often improves patient understanding and leads to a better-informed decision.

Military

In military and defense, mixed reality facilitates simulations, such as combat training, helping soldiers hone their skills in a real-world environment with minimal risk.

Mixed reality also aids strategic planning, providing 3D models of expected environments and situations. This also helps soldiers prepare to overcome expected obstacles.

Remote work

Remote teams employ mixed reality to foster more effective collaboration sessions and engagement with relevant products or documents.

Mixed reality also underpins digital workspaces for remote workers, delivering quick access to important virtual tools, which can be integrated into their physical workspace.

Differences between mixed reality and augmented reality

Although mixed reality incorporates elements of augmented reality, the two technologies have clear distinctions, including the following:

• Integration. The key difference between AR and mixed reality is the level of integration with the physical world. AR simply overlays objects into a physical environment. Mixed reality lets a user interact and engage with digital objects in the physical world.
• Technology. Mixed reality requires specialized equipment, such as a headset, to create an immersive experience. AR, used on common devices, such as smartphones or tablets, is more easily accessible than mixed reality.
• Uses. AR delivers fewer use cases than mixed reality. For example, AR is used primarily in mobile games such as Pokemon Go, as well as for marketing and retail purposes, such as virtually trying on a product. Mixed reality, as seen in the examples above, provides both a broader and more complex set of uses.

Mixed reality products and display technologies