MR Headsets Expert Roundtable – XR Today

MR headsets are facing increased interest this year. The Apple Vision Pro brought expected eyes to the technology, and late last year, Meta launched its Quest 3, which introduced its brand of MR at an affordable price following its Quest Pro launch.

Moreover, MR headset vendors are focusing on productivity usage – from frontline assistive reality tools to immersive desktop tools – and that could profoundly impact the future of work, as some believe MR headsets will become commonplace work tools like smartphones or laptops.

With the MR headset marketplace facing growth with other prominent vendors eyeing the space, considering the latest trends will help drive workplace adoption and understanding

To break some of the latest MR headset trends, XR Today compiled expert insight from:

• Said Bakadir, Senior Director, Product Management at Qualcomm Technologies, Inc
• Matt Tullis, VP of XR at Ultraleap

What are the biggest challenges facing the smooth integration of MR headsets in existing enterprise workflows and software?

Matt Tullis

A major roadblock to integrating MR headsets in existing enterprise workflows and software is a lack of compatibility, which can drive up costs and also compromise privacy.

While some XR software providers, such as Ultraleap, ensure their tech can be integrated with a variety of headsets, there are still players in the space contributing to a siloed environment — which puts end users (and innovation) at a disadvantage.

Both enterprises and consumers investing in XR want to be able to access their favourite apps/experiences on their headsets of choice, and while this seems simple enough, we’re not quite there yet as an industry.

MR headsets aim to mix spatial data with natural environments, as opposed to fully immersive VR headsets; what are this approach’s potential benefits and drawbacks?

Said Bakadir

Mixed Reality blends the virtual world into the real world by integrating virtual objects into your reality, bringing the best of both augmented reality and virtual reality together. Snapdragon MR technologies enable more immersive experiences to enhance reality with vivid colours and intuitive interactions.

Benefits:

Realistic immersion: By blending spatial data with real environments, MR headsets can create a more realistic experience. Users can interact with virtual objects while still being aware of their physical surroundings, resulting in a seamless integration of virtual and real elements. This level of immersion is ideal for hard skills and soft skills training, education, entertainment and wellness.

Context: MR headsets provide users with relevant information and virtual content that is contextually integrated into their real-world environment. This can be particularly useful in applications such as training, education, and remote collaboration, where users can interact with virtual objects in real-world contexts.

Awareness: With mixed reality, a sense of presence is reinforced. MR headsets allow users to maintain awareness of their surroundings. This can be beneficial in scenarios where users need to navigate physical spaces or interact with others while still engaging with virtual content.

Challenges:

Matching real environments’ spatial data with virtual content that is streamed right in front of the user’s eyes in MR headsets requires sophisticated tracking and mapping technologies to accurately align virtual content with the physical world.

Achieving accurate spatial mapping, low latency video see-through and realistic integration of the virtual with the physical in a comfortable device can pose technical challenges that need to be addressed for optimal user experience. Snapdragon XR platforms and our decades long R&D efforts around perception technologies allow us to face these challenges head on and with our innovations in processing and on-device AI.

Matt Tullis

Because they blend spatial data with real environments, MR headsets pave the way for more everyday use than VR can (realistically) allow. Users can still see the world around them — either via transparent lenses or camera input — while interacting with the digital world.

This brings essentially the same functionality as a smartphone into a headset, where users can access their favourite messaging and social media apps, get directions and even location-based recommendations while still being aware of their surroundings.

However, MR has room to grow. One particular area I see opportunity is in interaction, specifically hand tracking. While the Apple Vision Pro made great strides in hand tracking for VR/MR, many headsets today are still equipped with clunky hand tracking that isn’t as accurate as needed and requires users to make exaggerated, tiring and (if in public) unsafe hand and arm movements to navigate the virtual interface.

At Ultraleap, we’re making MR interactions more intuitive, accurate and easy to use with hand tracking that tracks subtle hand and finger gestures so users can control an MR device as easily as they scroll on their phone.

MR also has vast applications beyond traditional headsets. We’re working with automotive AR companies like Basemark to create AR-powered head-up displays, where drivers can use our hand tracking and mid-air haptic capabilities to control their infotainment system, navigate with interactive maps, and point and select objects of interest overlaid in the real world without averting their eyes from the windshield.

One drawback that is preventing many MR headsets today from becoming as ubiquitous and as well-used as smartphones is power. Frame-based sensors, which traditionally power hand and/or eye tracking on these headsets, tend to drain batteries, so they can’t be used for hours on end. Event-based sensors, however, such as the event-based hand and eye tracking system we debuted with Prophesee at CES in January, consume less power while providing low latency and increased privacy to users.

What integrated technologies most benefit MR headsets?

Matt Tullis

Hand tracking is now an essential element of the MR experience. Think about early MR/VR systems: those required controllers to navigate the virtual space, which can make the experience feel less natural and intuitive. Your hands, however, are the most natural controller of all. Hand tracking (and, as models get more sophisticated, even finger tracking) reduces friction in the MR experience and powers more intuitive digital interaction.

Ultraleap makes it easier for OEMs to apply this technology to their headsets. Our computer vision and machine learning models allow hand tracking to run on the same cameras used for SLAM and controller tracking, significantly reducing integration costs.

Additionally, adding haptics to MR experiences makes them even more grounding and immersive for the user. In real life, when you grab something or press a button, you can feel the experience. With haptics technology, you can get that same feedback in an MR/VR experience.

Said Bakadir

Several integrated technologies greatly benefit MR headsets including video see-through and perception technologies, and these are made possible with Snapdragon XR Platforms. A right sized, optimized processor is essential to meet the thermal and size constraints of MR headsets that also deliver optimal user experiences. For example, the Snapdragon XR2 Gen 2 was designed to solve these challenges and provide optimized MR experiences that are already accessible in Meta Quest 3.

Video See-Through: To avoid motion sickness and enable immersion, the video see-through (VST) pipeline needs to stream the user’s physical environment with high fidelity at a fast rate. For example, the Snapdragon XR2 Gen 2 and Snapdragon XR2+ Gen 2 include an optimized VST pipeline with 12ms average latency and image quality improvements for a more comfortable UX.

Environment Tracking: Perception algorithms like spatial mapping, object recognition, and segmentation create a digital map of the users’ space and anchor virtual content in a realistic manner. These technologies help with safety and peace of mind while a user is engaging with an MR activity.

User Tracking: Interaction technologies, including eye tracking, hand tracking, controller tracking, and 6DoF, track the users’ movement in space. All these algorithms require powerful on-device processing for optimal low-power, low-latency experience. Our perception technologies, packed in the Snapdragon Spaces SDK, provide the critical building blocks for next-gen MR experiences.

How are or should MR headset vendors ensure that devices are accessible to all users, regardless of hardware knowledge or physical limitations?

Said Bakadir

MR headset vendors can ensure that their devices are accessible to all users through promoting inclusivity and enabling all individuals to fully participate in the MR experience by considering the adoption of the following:

Universal product design: Having straps, corrective lenses, eye masks that adapt to face shapes, and head straps for a comfortable feel regardless of head size and hairstyle.

Multi-modal interaction inputs: Vendors can integrate multi-modal input methods to adapt to users’ preferences and/or abilities. This can include features such as voice commands, gesture recognition, or alternative input methods. By leveraging these technologies, users with physical limitations can effectively interact with MR content.

User Interface: Application developers can follow universal design and accessibility principles when developing for XR. MR headset vendors can provide options for users to customize the user interface based on their specific needs. This can include adjustable font sizes, colour contrast options, and customizable control schemes. By allowing users to personalize their experience, vendors can cater to different accessibility requirements.

Matt Tullis

The most effective solution for ensuring MR headsets are accessible to most users is to shift away from controllers and integrate advanced hand-tracking technology. For most people, hand tracking is the most natural and intuitive way of interacting in MR, as everyone knows how to use their own hands.

Hand tracking also takes away the friction for those that are less comfortable handling a controller or have disabilities that limit them from being able to use it properly. But it’s important to remember that not all hand tracking is created equal: some models require big, exaggerated movements to navigate an MR user interface, and for some users with physical limitations, that can be as inconvenient (or downright impossible) as using controllers. Vendors should ensure their models are capable of tracking smaller movements, which more users — regardless of age or ability — are capable of.

Eye tracking can also address physical limitations, enabling users to interact with the digital space without using controllers or their hands.