With their HoloLens 2 project, Microsoft and Volkswagen collaborate to put augmented reality glasses in motion

Imagine putting on a pair of augmented reality glasses as you head out in the morning and get into your vehicle. As your self-driving car takes you to your destination, you get holographic displays of traffic information, weather conditions, shopping recommendations and architectural highlights along the way. And as the day dawns, you can use holographic controls in front of you to adjust the interior temperature to your liking.

That’s the future of mobility envisioned by researchers at German automobile manufacturer Volkswagen, who see augmented reality as one of the key components of future mobility concepts. To get a little closer to that vision, Volkswagen collaborated with Microsoft to enable the HoloLens 2 mixed reality headset to be used in moving vehicles for the first time.

The new “moving platform” mode for HoloLens 2 overcomes a major limitation of mixed reality headsets and creates potential for the technology to be used in new ways — training drivers to handle challenging road conditions, for example, or creating new user experiences for autonomous vehicles. And while mobility is Volkswagen’s focus, the capability could in future be shared across other industries.

“We think mixed reality information is the most intuitive information we could provide to enhance our customers’ user experience,” says Dr. Andro Kleen, head of the data science team at Volkswagen Group Innovation. “Because what you see there, and what you need to process, is very close to what humans normally see and process. It’s not so abstract.”

An early adopter of augmented reality technology, Volkswagen introduced an augmented reality head-up display in its ID. family of electric cars in 2020 that projects navigation arrows, lane markings and other information onto the environment in front of the cars.

But Volkswagen was thinking about the potential of augmented reality even earlier, Kleen says. In 2015, for instance, Volkswagen embarked on a research project using self-piloted vehicles and augmented reality to teach driving on a racing circuit. Tested at Volkswagen’s track facility in Ehra-Lessien, Germany, the Race Trainer system used a head-up display that superimposed lines of arrows on the track for drivers to follow, and provided steering and braking cues to guide them through a gradated set of lessons.

Volkswagen hoped to use HoloLens for the research project, but soon ran up against a problem. When the device was put into a moving vehicle, its sensors lost tracking and the holograms it normally displays disappeared. Kleen’s team approached Microsoft for help and connected with Marc Pollefeys, Microsoft director of science and an expert in 3D computer vision and machine learning. The Volkswagen team traveled to Microsoft’s headquarters in Redmond, Washington to show Pollefeys and his team its work with HoloLens and its vision for augmented reality technology.

“We had extensive discussions,” says Pollefeys, now the director of Microsoft’s Mixed Reality and AI Lab in Zurich, Switzerland. “They presented their use cases and what they were hoping to enable. They were eager to work with us to find a solution and be able to use HoloLens in those situations.”

The two teams began collaborating around 2018 to develop the moving platform feature for HoloLens 2, which required solving a fundamental problem. HoloLens uses two main types of sensors that measure its motion — visible light cameras and an inertial measurement unit, or IMU, that gauges acceleration and rotation speed. Together, the sensors mimic how humans see and move through the world.

But similar to how being in a car or boat can cause motion sickness when what appears to be a stable environment is actually moving, when the tightly coupled HoloLens sensors start disagreeing in a moving environment — with the inertial measurement unit recognizing motion and the cameras unable to — things fall apart.

To address that issue, Pollefeys’ team developed an algorithm that models the discrepancies between the sensors and allows HoloLens to continue tracking. But testing the capability posed another challenge. Since Covid-19 restrictions in Switzerland made it impossible to try out the technology in a vehicle with another person at the time, Joshua Elsdon, a Microsoft senior software engineer who worked on the project, had to find solutions from his Zurich apartment.

Elsdon devised a mock platform using a plastic box, sticking bits of tape inside to add visual texture and give the HoloLens cameras elements to track. He rode trams and buses around Zurich wearing a HoloLens headset, making sure its holograms held up as the vehicles moved. At night, Elsdon rode up and down elevators in his apartment block, testing the technology.

“We had to do a lot of testing in my apartment. These aren’t ideal development conditions,” says Elsdon, who is now based in Redmond. “All of this stuff was done remotely and distributed across different countries, which was interesting.”

After the initial prototyping was done, the team also conducted testing on Puget Sound, near Microsoft’s Redmond campus. They rented recreational boats, took them out on the water and used external measurement equipment to assess the performance of HoloLens’ head-tracking system in a moving platform.

Microsoft later tested the feature with Volkswagen. The Volkswagen researchers established a bidirectional data connection between the vehicle and the HoloLens in order to display and control real-time information from the car. Finally, the team implemented several demo use cases investigating how virtual interfaces could enhance the interior of future vehicles.

“We connected a positioning system that tracks the location of the vehicle. This way we were able to also place 3D elements such as information on point of interests outside of the car. This opens up completely new possibilities to not only display holograms within the driver’s forward-facing field of view, but also wherever the user wearing the glasses is looking,” says Michael Wittkämper, augmented reality expert at Volkswagen.

Microsoft rolled out the moving platform feature a few months ago and it is already attracting interest from maritime companies and organizations, which have been using HoloLens to remotely connect maritime workers with mechanical experts through Microsoft Dynamics 365 Remote Assist. The app allows an expert in another location to look through the other person’s HoloLens 2, share their field of view, diagnose a problem and provide input.

Previously only usable when a ship is in port, the capability is even more needed when vessels are at sea and a piece of equipment breaks down.

“The more remote the equipment or machine is, the harder it is to get the expert on site,” says Pollefeys, who is also a professor of computer science at ETH Zurich, a public research university. “This feature turned out to be critical to unlock HoloLens 2 for the maritime space.”

HoloLens’ moving platform feature is currently supported for use on large ships, and Microsoft plans to further refine it for use in elevators, trains, cars and other moving environments. Kleen envisions multiple ways the technology could be used to connect the inside of vehicles with the outside world, such as helping truck or bus drivers navigate through narrow streets, identifying points of interest along driving routes or providing entertainment for passengers.

Kleen and Pollefeys believe augmented reality will become an increasingly important aspect of mobility in the future as smaller, more compact versions of smart glasses become available. Pollefeys characterizes mixed reality glasses as the third generation of personal computing devices, after personal computers and mobile phones.

“Microsoft is not only interested in commercial devices like HoloLens, but also in the longer term, in devices that would make sense for consumers to use in daily life,” Pollefeys says. “With augmented reality glasses, you could walk around the world and information can appear in context, where it’s relevant. You could communicate with other people and also in 3D, as opposed to on a small screen.”

Kleen imagines augmented reality enabling seamless and connected mobility experiences, with people donning smart glasses as they leave home and receiving information through them, from navigational assistance to entertainment, as they travel through their day.

“We think of this as moving toward a mobility system where different products and mobility solutions will be connected,” Kleen says. “The basic assumption is that this technology will become lighter and smaller, and we think that as that happens, more people will get their hands on it and integrate it into their daily lives — and thus into their way of moving from A to B.”

Top photo: Volkswagen demonstrated the future of fully autonomous driving in its Gran Turismo concept car, the ID.VIZZION. (Images courtesy of Volkswagen)