What Is Mixed Reality? – The Technology Explained17 min read31/08/2018
Mixed reality, also known as hybrid reality, is a next-generation technology that combines both augmented (AR) and virtual reality (VR) to generate a new semi-essential environment and visualization where physical and digital objects co-exist and interact in real time. You could see this technology as holograms in fantasy movies like Iron Man 3.
With the growing popularity of VR and AR apps, the mixed reality (MR) market is expected to show the significant growth in the next few years. The Allied Market Research forecasts the MR market to reach $5,362.1 million by 2024 with a compound annual growth rate (CAGR) of 71.6% from 2018 to 2024 compared to only $123.2 million in 2017.
Mixed reality is a technology that involves placing virtual objects into the real world and adjusting them to a physical environment in order to expand its functionality as well as allow users to interact with both virtual and real objects in real-time. Experiencing MR currently means interacting with high-quality computer-generated objects in a form of holograms using special headsets like Microsoft Hololens.
The main sense of the technology is creating realistic 3D objects that would be perceived by users as a part of the physical environment. It’s unlike augmented virtuality where real-world objects should adjust a virtual environment.
Augmented virtuality is a concept that implies inserting physical objects into a virtual environment, unlike augmented reality where the system overlays the real-world environment with digital 3D objects. You can think about augmented virtuality as a way of, for example, planning the redesign of the apartment with current home appliances or particular furniture.
By adding his or her TV to the computer-generated environment, a homeowner can visualize how a specific room would look like with a completely different design and match current appliance. This feature can be useful for interior design agencies whose clients may need to remodel their houses without the need for purchasing new appliances.
A size, shape, color, orientation, and physics are those factors that influence how we see real objects. To create a highly realistic MR experience, developers rely on how human eyes and brain perceive the real world. This information allows programmers to make users perceive virtual objects as physical ones within the physical environment.
To trick human eyes in perceiving computer-generated objects as physical, mixed reality headsets use special scanners that collect data about the real environment in a form of three-dimensional images and sensors that determine the orientation of the user’s head. This data enables MR systems to define the proper place for virtual objects within the physical environment.
How does mixed reality work?
MR provides users with a capability to experience both a computer-generated and physical environment seamlessly in real-time. Mixed reality works by scanning a real-world environment and generating a 3D map of user surroundings. Using space coordinates, MR apps place virtual objects within the real world in accordance with the user’s position and his or her head orientation.
Thus, wherever users move and no matter where they look while wearing a mixed reality headset, virtual objects become larger or smaller thus seeming closer or further in the real environment. In MR, users also can look at digital objects from different angles and perspectives.
Mixed reality combines the characteristics of virtual and augmented reality. Like VR, MR provides virtual content as well as makes users perceive it as a real world. Like AR, mixed reality adds another digital layer to the physical environment. Combined in a single headset, these features form highly realistic scenarios that significantly expand our reality.
To properly place virtual content and enable users to interact with digital objects, the mixed reality technology uses different types of input data. This includes a gaze, user gestures, data received from motion controllers, and voice input data.
MR relies on a gaze vector to determine what users are looking at within the physical environment. The orientation of a user’s head allows a mixed reality system to understand the real-world environment as well as the right angle for displaying virtual content. In addition, the gaze direction enables the system to properly move those digital objects users intent to interact with.
With hand gestures, users can manipulate 3D virtual content in mixed reality. Each gesture corresponds to a specific command which the system can recognize. A set of available gestures vary from headset to headset.
Basic Microsoft Hololens gestures:
- Air tap. A basic gesture that enables users to select an option is an air tap. This gesture is similar to a typical mouse click.
- Bloom. To return to the home screen, you can use the bloom gesture. Raise your hand with the fingertips together and then open it.
- Tap and hold. With the tap and hold gesture. you can select and drag files and options using Hololens.
- Manipulation. To make the MR system move a hologram in the way you need it, for example, rotate or resize it, you can use intuitive manipulation gestures in the air. These gestures are useful when using drawing apps while wearing Microsoft Hololens.
- Navigation. To navigate through the virtual menu, you can use various navigation gestures like scrolling or swiping.
The Windows Mixed Reality system relies on motion controllers to receive input data from users. The main advantage of this hardware over gestures is that it has a precise position in space, thus ensuring the accurate response to user actions.
Besides gestures and motion controllers, Hololens can receive input data by recognizing specific voice commands. This enables users to navigate through nested menus in a fast and simple way. To select an option, you should first gaze at your hologram and then pronounce the “Select” voice command.
Mixed reality software includes two types of solutions: development and application software. Application software is a solution that enables users to experience mixed reality through MR devices. To create MR experiences, developers use software development kits. These tools help programmers develop realistic virtual content for specific mixed reality devices. To create MR experiences for Windows Mixed Reality headsets, developers can use Visual Studio with the Windows 10 SDK. When working on MR apps, programmers use a set of software components that make the proper hologram placement possible.
Image recognition is a software feature that allows computer systems to accurately analyze images and identify them. While simple augmented reality applications can recognize QR codes to determine where to place virtual content, MR apps rely on more complex techniques like image recognition to identify objects within the physical environment.
Simultaneous localization and mapping (SLAM)
To properly function in unknown or dynamic environments, mixed reality systems need to understand these environments. For the technology, understanding the environment means having a digital map of this environment as well as locating and tracking moving objects within it. To do so, MR relies on simultaneous localization and mapping also known as SLAM. To determine the user’s position in the physical environment, the SLAM feature uses real-time depth sensor and image recognition.
Another feature that allows mixed reality systems to understand physical environments is spatial mapping also known as 3D reconstruction. This feature creates a detailed 3D map of the real-world environment around MR devices. By properly combining the physical environment with the virtual one, mixed reality apps can generate highly realistic holograms.
To be perceived like real, holograms generated by MR devices should be properly placed within the physical environments. Therefore, they should be accurately adjusted to real-world objects with the right positioning and orientation. This becomes possible with spatial coordinate systems that enable mixed reality apps to determine where to place holograms.
Placing a hologram is a dynamic process rather than one complete task. Users may manipulate a hologram or move within a room. That’s why the hologram should adjust to user movements. MR apps use spatial anchors to monitor proper hologram placements. You can think about spatial anchors as points in space. Mixed reality solutions place virtual objects in correspondence to these anchors and continuously track their relative position.
In the real world, you can hear a car getting closer because the sound it generates gets louder. You can also tell what direction it is coming from. Spatial sound makes the virtual content in the form of holograms be perceived more naturally by users. 3D sound coming from specific directions seem real as if it was coming from real objects. This makes an MR experience more immersive and natural. Spatial sound enhances the user’s perception of virtual objects.
Mixed reality apps run on special devices called MR immersive headsets. These headsets consist of a wide range of components and sensors that collect data from physical environments as well as enable software solutions to analyze and understand real-world surroundings in real-time.
MR immersive headsets have semiconductor components that include active, passive, and electromechanical elements. These components make transferring electromechanical signals possible within mixed reality hardware systems.
MR headsets contain a set of various built-in sensors that collect data from physical environments. Head-mounted displays may rely on six degrees of freedom (6DoF) inertial measurement units (IMUs) that allow mixed reality solutions to properly combine the digital content and real world in accordance with movements of user’s head (up and down, right and left turns, the angle of right and left lists). These units also enable users to interact with virtual objects. 6DoF IMUs consist of various sensors described below.
A magnetometer is a sensor that determines the direction and strength of the magnetic field. These sensors can be divided into scalar and vector. While a vector magnetometer measures magnetic fields in the specific direction, a scalar sensor measures their strength. Mobile devices use magnetometers as a means of a digital compass.
Accelerometers are used to measure non-gravitational acceleration. This includes the motion, rotation, and motion gestures. Accelerometers can track motion either in only two directions by 2 axes X and Y or in 3 directions (X, Y, and Z). These sensors help mobile navigation apps accurately display a user’s position on the digital map when moving and calculate the time needed to reach a destination.
Gyroscopes are sensors that measure the orientation in space. They allow mobile navigation apps to show in which direction a user is looking. These sensors also enable tablets and smartphones automatically rotate images on their displays when in the horizontal or vertical position.
You might have noticed that your smartphone automatically blocks the screen during a phone call. Even though the call is active, it unblocks the display once you take the device off your ear. This becomes possible with proximity sensors. They detect physical obstacles within a specific range by transmitting infrared impulses. This principle is similar to the one how LIDARs, radars, and sonars work.
MR device may have other sensors and use them upon software request. These sensors include pressure, position, 3D, and fiber optic sensors.
MR head-mounted displays use Li-ion batteries as a means of a power source. Microsoft Hololens has a battery of 16,500 mAh. This is enough for 5.5 hours of average use or around 2 weeks in the standby mode.
Mixed reality head-mounted displays (HMDs) provide users with virtual content by projecting 3D images on a semi-transparent display. These images further reflect from the display to the user’s eyes through a beam-splitter.
Microsoft HoloLens is a mixed reality headset that uses the 64-bit operating system Windows Holographic (Windows 10). Users can control Microsoft Hololens using a Hololens Clicker or voice commands. According to The Verge, Microsoft is working on the next edition of their HMD. HoloLens Sydney will have a wider field of view and lighter weight as well as will be cheaper than the current device which costs $3,000.
Samsung Odyssey has two 3.5-inch AMOLED displays that ensure high-resolution images. The headset provides high-quality 360-degree spatial sound through its built-in AKG headphones. The price of Samsung Odyssey is $417 with motion controllers included.
Acer Windows Mixed Reality
Acer was among the first vendors that had presented their mixed reality headsets. Their head-mounted display has two displays with the 1440 x 1440 resolution. One of the features of Acer’s HMD is that users can flip up the screen to get back to the physical environment. It offers a display resolution of per eye. The price of this device is one of the most affordable on the market. It costs only $199 with wireless controllers included.
Lenovo Explorer Mixed Reality
Lenovo Explorer is an MR headset that supports the Windows Mixed Reality platform. It has two 2.89-inch displays with a resolution of 2880×1440, 90 frames per second, 110-degree field of view, a set of sensors, two movement tracking cameras, a pair of motion controllers.
Despite the growing popularity of the mixed reality technology, companies and organizations hesitate to deploy MR solutions. According to VRfocus, customers have purchased only 50,000 Hololens headsets as of May 2018. Furthermore, the technology itself faces some challenges on the way to the wide user coverage.
High hardware requirements
To provide users with a fully immersive experience, MR devices should have the minimal latency when it comes to real-time processing. The delay in tracking changes of dynamic physical environments and placing virtual content leads to destroying the illusion of the continuity of real and digital worlds.
The thing is that mixed reality apps require a lot of processing resources. Even powerful computers that easily support popular video games may not support MR software. To eliminate any possible lags, your computer should have at least the Intel Core i5-7200U processor, 8GB of RAM, free 10GB of disk space, and the Nvidia GTX 1050 graphics processing unit (GPU).
Lack of realism
Despite hardware has enough capacity to support mixed reality without any lags with video of the 60 fps frame rate, this is not a guarantee that users will perceive the digital content as real one. Virtual 3D objects also should be designed in the way to be perceived natural even though they don’t replicate the appearance of existing objects.
The graphics in video games has turned from flat 2D images into deep three-dimensional high-resolution objects that replicate the physics of the real world. While today’s console-based video games have the graphics looking almost like a movie, reality technologies are still far from the realistic image. However, it’s just a question of time.
Mixed reality is not all about gaming. It has a wide range of possible use cases as well as VR and AR. According to Statista, about 53% of U.S.-based mid-market commercial organizations used MR for employee training purposes as of July 2017.
Possible mixed reality use cases:
- Remote assistance. Remote experts can help trainees or inexperienced workers solve a problem in real-time using See-What-I-See (SWIS) MR glasses
- Training. Using a mixed reality immersive headset, trainees can learn by doing without a risk to damage property or injure other people. Students can always get their virtual step-by-step instructions or manual with a gesture, voice command, or push on the button.
- Design. Industrial designers can make corrections in real-time and modify their virtual mock-ups in space without the need for manufacturing a physical model.
- Virtual 3D objects in offices. Like designers, office employees can work on digital three-dimensional content moving from displays to MR environments.
- Holographic meetings. With a video conference, you can the faces of your interlocutor on a display of your computer or mobile device while talking to this person. With the mixed reality technology, you can take a step further and see your colleague or friend right in front of you in the form of a hologram.
- Data analysis. With an MR headset, you have access to valuable data and charts at any time regardless of how far you are from your computer. Furthermore, you can access and manage this data in a hands-free manner using voice commands.
Let’s now discover how various industries have already started deploying mixed reality.
During a medical intervention, surgeons need a constant access to patient data such as laboratory test results or X-ray examination images as well as keep an eye on the patient’s condition in real-time: heart rate, blood pressure, etc. All this involves distraction from the intervention every time a surgeon needs to check these measurements.
Furthermore, there can occur situations when a surgeon may need to consult with colleagues about an issue happened during the intervention. In this case, the choice of colleagues is limited by those who currently are in the operating room. To help surgeons improve their medical operations, Liverpool-based Alder Hey Children’s Hospital decided to implement mixed reality technologies. The medical organization will equip their meeting rooms with Microsoft Surface Hub displays and allow surgeons to use Hololens HMDs.
These widescreen displays will enable medical employees to effectively communicate with each other while staying on different continents. Physicians will be able to remotely exchange patient health information such as test results rather than receive them on paper which takes a lot of time. With Microsoft Hololens, surgeons will have access to up-to-date patient health data in real-time. This will allow them to focus on the surgery and timely react in the case of unexpected change of the patient’s health condition.
A typical vehicle designing process involves several stages: drawing a future design on a paper, building a physical life-sized clay mockup, and developing a digital model using special software like Autodesk. The cost of creating a clay car model varies from $100,000 to $300,000. In addition, this is a time-consuming task.
Mixed reality has the great potential in the automotive industry, especially as a means of a design tool. With this technology, designers can work on the exterior of vehicles in a more efficient way and interact with digital life-sized car models as well as quickly apply changes.
In its Design Center, Ford uses Hololens to create to improve collaboration between designers, increase productivity, and reduce time to market for their new cars. Ford designer use mixed reality along with clay models to see how a particular car part would look like after specific modifications instead of modifying the initial clay mockup.
While investigating scenes of crime, the police mark evidence or other important things that can help with further crime solving, Once criminalists examine evidence and take photos of the scene, they clean it since, for example, blood stains can remain forever in public places. They terrify people. Therefore, those photos are the only crime stamp that can be used to recreate the scene of crime if criminalists haven’t taken something important into account.
That’s why Black Marble decided to develop a mobile app for Microsoft Hololens. The Scene of Crime app will enable police officers to efficiently examine crime scenes, collect important data for further investigation, and mark evidence with digital markers. This will keep the crime scene untouched, thus significantly facilitating the investigation. With the mixed reality app from Black Marble, police officers will be able to virtually reproduce the scene of crime even if all traces have been already ruined. Thus, investigators can virtually travel in time and investigate digital markers and all pieces of evidence like they were right after the incident happened.
The company’s Command and Control app for HoloLens is one of the mixed reality examples of how the technology can improve communication between employees. With this mobile app, a police officer staying at the police station can remotely keep an eye on the situation on the street along with patrol members thanks to a SWIS MR headset. Wearing a head-mounted display, a patrol policeman can communicate with his chief or dispatcher while providing his colleagues with real-time streaming video from the streets. Thus, those policemen in the command center can timely react to emergency situations when their colleague needs urgent backup.
Mixed reality combines the advantages of both artificial and real worlds and can provide users with valuable data in real-time while enabling them to have their hands free. While wearing an immersive MR headset, users can share what is going on around them with their team members. They also can examine virtual 3D objects from any side and interact with them while keeping control over the physical environment.
With these capabilities, the technology can bring significant benefits to various industries like automotive, healthcare, and military. By leveraging capabilities of both VR and AR, mixed reality wipes out the border between virtuality and reality as well as enables people with brand new capabilities previously impossible in the physical world.