Mixed Reality Use Cases and Challenges

Since the mid-1950s, or maybe even before, humans have been on a mission to blur the lines between physical and digital realities. Combing both real and virtual worlds have the potential to create new opportunities and make something completely unique.

Immersive technologies aim to extend our concept of reality through Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). These innovative technologies are slowly becoming mainstream through video games, conference apps, and more.

Of all immersive technologies we have today, Mixed Reality is probably the least explored. Although we’re scratching at the surface of MR with AR technologies, there’s still plenty of work to do before we engage in fully optimised MR experiences.

But before diving right into MR, let’s take a step back and get some definitions out of the way.

What are Immersive Technologies?

The term “immersive technology” describes the process of creating, displaying, and interacting with applications, content, and experiences in a new way. By combining AR, MR, VR, and haptic technologies like a “3D touch” or “kinesthetic interaction,” immersive technologies have transformed digital experiences.

By bringing together the users’ sight, touch, and sound, people can feel part of a simulated artificial environment. Creating these experiences requires state-of-the-art hardware (like VR headsets) and software.

According to Juniper Research, by 2024:

• 67% of smart glass apps will focus on gaming and multimedia.
• 75% of MR apps will be delivered through smartphones.
• 40% of MR apps will target social media.

By the end of 2024, the AR and MR market is expected to be worth around $43.8 billion.

What is Mixed Reality?

Although MR is definitely the rising star of immersive technologies, it’s also the least understood. The confusion often arises because MR combines the key benefits of both AR and VR and transforms them into one incredible experience.

According to Microsoft, MR or Mixed Reality is a fusion of both physical and digital worlds that will help unlock natural and intuitive 3D computer, human, and environmental interactions. It’s possible today because of recent advancements in graphical processing, computer vision, cloud computing, input systems, and display technologies.

It’s important to note that the application of MR has gone beyond displays to include:

• Collaboration on 3D assets in MR environments.
• Environmental understanding (including spatial mapping and anchors).
• Human understanding (including eye-tracking, hand-tracking, and speech input).
• Location and accurate positioning (in both physical and virtual environments).
• Spatial sound.

Although the evolution of MR has been slow to date, don’t expect it to always be this way. This is because MR liberates us from screen-bound experiences. It also opens the door to instinctual interactions with data, our physical environment, and our friends.

How Does Mixed Reality Work?

The way MR works is quite similar to AR. However, its applications have evolved in recent years because of the development of smart systems and tools that enable deeper interactions between humans, computers, and objects.

MR depends heavily on the rapidly evolving human and machine relationship. To work, MR tools and applications have to understand different human actions within a specific digital space and the surrounding landscape.

With the help of multiple cameras, sensors, and Artificial Intelligence (AI) technologies, we can process massive amounts of data about a specific space and leverage that information to deliver digitally-enhanced experiences in near real-time.

For example, whenever we wear a pair of MR glasses, the cameras and sensors in those glasses get to work collecting and distributing data to an application that is basically creating a virtual map of our physical world. This approach enables the creation of holographic images and content we can include in our reality through image projections.

For MR technologies to work effectively, they need to be able to track:

• Ambient sounds and lighting (for added realism)
• A user’s body movements and position
• Objects and physical locations
• Objects surfaces and boundaries (through spatial mapping and scene understanding)

Cloud computing enables the rapid processing of advanced input sensing. At the same time, environmental perceptions enable MR applications to merge our physical world with a virtual one successfully. We can do this in a manner that goes far beyond the means of AR technologies.

Mixed Reality Use Cases Across Industries

MR can break away from AR-related limitations to closely connect both virtual and real worlds. Let’s explore some MR use cases to see what’s already possible with this cutting-edge immersive technology.

1. Manufacturing

In an industrial setting, companies are experimenting with digital twinning. Digital twinning describes the method of building highly realistic models of systems, processes, products, and even machines for development, testing, and validation.

For example, leveraging MR, digital twins help provide technicians with the information they need to simulate and resolve different technical issues. Although digital twinning was pioneered in manufacturing, it can be adapted to any industry. We can also apply digital twinning to cross-industry applications use cases like infrastructure and automation.

2. Communication

When we use MR communication protocols, it essentially annihilates present Zoom experiences. Instead of blurring your background or sharing your screen, you can engage each other in digital spaces and immerse yourself in the subject matter.

So, instead of talking in front of static wallpaper, you can actively share your messages using images, sound, and emotions to get your point across in a highly engaging manner.

Multidisciplinary teams can also work together and collaborate more effectively in dedicated three-dimensional digital spaces.

3. e-Learning

MR can take online learning to the next level by showing you what you’re learning about in 3D. This approach will help students form solid memories and better understand what they are learning about.

We can also apply this same idea to health and safety and industrial repair and maintenance training. This method helps students learn using a pseudo “hands-on” approach without the risk of injury.

3. Retail

MR can help blur the lines between e-commerce and brick-and-mortar retail experiences. For example, you can use a retail mobile application to identify all the products in a store automatically and add them to your shopping cart, leveraging computer vision and machine learning capabilities.

These products can be purchased or bookmarked to buy later or even to share with friends. In this scenario, when you wear a pair of AR glasses, you can quickly see bookmarked products, reviews, and more inside a store. MR, in this case, can also help shoppers promptly identify and grab what they need in the store.

4. Marketing and Customer Service

MR in marketing has the potential to promote products in a way we have never seen before. Using high-tech experimental marketing techniques and studying individual engagement patterns, businesses can automatically engage each potential customer across multiple touchpoints.

For example, a customer can immediately go to a virtual branch, engage a virtual customer service representative, and learn about the product (by touching and experiencing the product virtually). Brands can adapt the virtual representative’s responses based on the customer’s preferences and transaction history.

5. Healthcare

MR in healthcare is fascinating (to say the least). In fact, MR is about to revolutionise medical education by providing valuable “hands-on” experiences. It can even evolve into the initial training ground for next-generation surgical robots learning autonomous steps within a surgical procedure (for example, suturing).

Other potential benefits include rapid diagnosis by superimposing information (such as patient history and images) related to the patient’s condition. In fact, the next time you go to see a doctor, they might be wearing MR glasses, ready with an instant diagnosis.

In pre-and post-surgery assessments, MR can optimise outcomes through personalised interventions driven by data analytics. Remote surgery is already a reality. Soon, surgeons with MR headsets in different locations can also assist in live surgeries.

MR also offers surgical staff new avenues to evaluate information that’s compatible with surgical workflows. For example, a surgeon can receive real-time guidance in their field of view during an active surgical procedure. Similarly, MR glasses can project real-time data like the patient’s heart rate and blood pressure.

Finally, real-time 3D views of anatomy can provide additional details to help doctors and medical staff make better-informed decisions. As MR in healthcare matures, it has the potential to turn some of the riskiest medical procedures into a walk in the park.

The Challenges Ahead

Although MR development is currently going through a period of significant acceleration, some obstacles remain. However, the primary challenge is to overcome the fear of change. Other times it can also be the IT department’s lack of understanding or expertise. 

However, once businesses realise the actual benefits of MR, convenience will override paranoia. But even then, MR will have a long way to go before it achieves large-scale adoption.

Technical Challenges

MR hardware has evolved rapidly and become more affordable. But it remains in its infancy stage with limited battery life, discomfort when worn for an extended period, and a relatively small field of view. However, these challenges won’t last and will be resolved sooner rather than later.

We also need more tools to support hands-free interaction and collaboration in digital spaces. Although speech is one way of overcoming this challenge, it can be unsuitable in a learning environment.

Latency and Accuracy

Latency has been an ongoing problem for VR and AR protocols. So, it’s safe to assume similar issues will plague MR.

Maintaining anatomic accuracy while producing graphically palatable images in a healthcare setting will be challenging. We can’t really have MR in the operating theatre until we overcome this hurdle.

Final Thoughts

We’re on the cusp of living in a multilayer reality where lines between the real and virtual are almost impossible to tell apart. As the industry continues to innovate and accelerate, we can count on efficient business operations, improved creativity, superior healthcare, and enhanced customer experiences.

5G is already helping companies reduce latency, and this will have a positive impact on MR. Furthermore, as technology becomes lighter, faster, and more affordable, you can bet that we will see a Mixed Reality future sooner than anyone expected.

Credit: https://www.immersivelearning.news/2023/03/15/mixed-reality-use-cases-and-challenges/