HoloLens 2 brings new immersive collaboration tools to industrial metaverse customers
Even with precise written instructions, some parts of the automotive assembly process still require a bit of art, says David Kleiner, who leads Toyota Motor North America’s Applied Technology Research Lab.
Take hammering on a door edge guard. That’s the thin strip of steel that protects against dents and dings when you fling the door open into a wall or parked car. Use too much force, and a worker can damage the vehicle. Too little, and the edge guard will pop off the next time the door gets slammed shut.
Recently, a worker at a Toyota Logistics Center in New Jersey was struggling to find that balance. So they put on a HoloLens 2, Microsoft’s mixed reality headset, opened a holographic window and called a colleague in California for help. They were instantly looking through the eyes of someone on the opposite side of the country.
Here’s the trick, their coworker said: Listen for this sound. And they struck the piece at just the right angle, producing a distinct tone and a clear visual that the colleague could see and hear as if they were right next to each other.
That sound of success is now helping other Toyota workers install door edge guards. The clip was recorded on a HoloLens 2 and captured in the step-by-step holographic instructions in Microsoft Dynamics 365 Guides, giving all workers the benefit of that wisdom.
For Kleiner, that story shows how HoloLens 2 is a tool perfectly in tune with kaizen, a core Toyota principle of continuous improvement. And it explains why the device has sped out of Toyota’s innovation labs and into its everyday workplace. Workers at six U.S. Toyota Logistics Centers use HoloLens 2 for hands-free training, guidance and collaboration. The headsets will roll out soon to sites in Canada and Mexico, Kleiner said, with more expansion planned.
“The killer metric for Toyota is speed,” Kleiner said. “The faster we can train people and solve problems, the faster we can get a product to market. That’s why we want to overcome location, we want to overcome time, and we want people to move faster and share knowledge. HoloLens has enabled us to do all that.”
Over the past year, Microsoft has made significant investments in the HoloLens 2 platform in response to accelerating adoption from enterprise customers like Toyota, from new updates enabling immersive collaboration to solutions that allow companies to scale and manage a fleet of devices. They’re benefiting workers who wear the headsets across industries, whether in hospitals, semiconductor clean rooms, power plants, anatomy labs or construction sites, the company said.
Microsoft’s latest wave of updates is combining its key Dynamics 365 Mixed Reality Apps and adding Microsoft Teams and OneDrive to existing integrations with Azure and Power Platform. As a result, Microsoft is delivering the cloud-powered productivity and collaboration tools that deskbound workers have long had at their fingertips to the frontline worker. No matter where they are, HoloLens 2 users can summon an array of holographic windows with a Teams call or chat, a Power BI dashboard, a Word document, a PDF or video, their OneDrive folder, or their calendar and operate in an immersive, 3D experience.
“We’re truly delivering Windows in mixed reality,” said Alysa Taylor, Microsoft corporate vice president for Azure and Industry.
Those capabilities help explain why businesses like Toyota have moved beyond just kicking the tires on the metaverse.
For many, that word brings to mind avatars that represent us in online worlds. It’s a partly accurate picture, Taylor said. Microsoft sees several different metaverse categories. There is the familiar consumer metaverse, where we shop and play with friends and family. And there are the commercial and industrial metaverses where we collaborate with colleagues and get work done.
The industrial metaverse that Toyota and others are exploring is a fundamentally new way for humans and AI to work together to design, build, operate and optimize their physical systems, Taylor said. With Microsoft Azure, Dynamics 365 and mixed reality offerings that bridge the digital and physical, customers can build digital twins of a factory floor or warehouse and simulate manufacturing or supply chain processes in the cloud. That allows them to refine those processes in the industrial metaverse – whether to boost operational efficiency or shrink their environmental footprint – before committing them to physical form.
Mixed reality is a key technology of Microsoft’s industrial metaverse solutions, Taylor said. HoloLens 2 delivers those solutions all the way to frontline workers who work with their hands and can’t be tethered to a computer or keyboard. Since its launch, Microsoft has incorporated customer feedback and invested in making the devices truly work on the frontlines – from longer lasting batteries to designs that fit over safety glasses to updates that make it easy for a fleet of devices to be managed by a customer’s IT department.
“Frontline workers form the backbone of many of the world’s largest industries, yet they’ve been largely underserved by technology,” Taylor said. “So much knowledge and information is in the cloud, but how does that deskless worker in the field or on the factory floor access that digital world?”
Microsoft’s answer is HoloLens 2 and its Dynamics Mixed Reality suite of apps. With Azure cloud services doing much of the heavy lifting, they can take digital information and integrate it into the user’s physical world. That holographic experience helps them learn industrial processes faster or support colleagues around the globe with expert advice as if they were in the same room, without the expense or environmental impact of long-distance travel.
“These employees don’t have desks; giving them a laptop just won’t work,” Toyota’s Kleiner said. “We want HoloLens to be our screens for our frontline workers. When they’re wearing a HoloLens, they now have a screen that gives them all the digital tools they need.”
Driven by customer feedback
Since HoloLens 2 launched in 2019, Microsoft has rolled out 34 monthly software updates. Some delivered support to new regions. Many added new functionality, like a color-blind mode to improve accessibility or tweaks to the head-tracking algorithms so customers can wear HoloLens 2 in unstable environments, like on a ship.
Every single one has been driven by customer feedback, said Scott Evans, Microsoft’s vice president of Mixed Reality.
“Heads-up, hands-free technology enables big leaps forward in how people work,” Evans said. “But often that means going through a cycle of doing a pilot, measuring and learning, and then scaling up the deployment. We engage customers throughout that journey, and it’s a big learning process for us.”
The latest update delivers a top customer ask: bringing together Microsoft’s two key Dynamics 365 Mixed Reality apps. The first is Dynamics 365 Guides, which takes users through step-by-step holographic instructions. The second has been the most popular scenario for HoloLens 2 users: Dynamics 365 Remote Assist.
The cross-country calls about door edge guards showed the power of Remote Assist, which lets a frontline worker virtually bring collaborators into their physical space. In this “see what I see” scenario, those colleagues can be anywhere in the world, on any device. They just get a Teams call, and from there they can now annotate in 3D space and augment what that frontline worker is seeing. Combining the apps was a priority request from Toyota, which has been delivering feedback to Microsoft since the original HoloLens launched in 2016. The updated Dynamics 365 Guides delivers a much better experience, Kleiner said.
“Someone can grab a HoloLens, start a Guides session, and literally have a trainer in their head,” he said. “If they do need help, they can call an expert right from the app.”
Toyota feedback also helped improve other aspects of the Guides experience. For example, technicians used to put a QR code on the hood of a vehicle, scan it, and then follow the holographic instructions.
But those holograms could appear to drift as a worker moved around the car. Toyota worked with Microsoft to develop “object understanding.” That meant Toyota technicians could scan the entire vehicle, which helped lock a hologram in place and eliminate the parallax problem. A hologram pointing toward a bolt would always point to the exact location, no matter the viewer’s angle.
Over the years, the authoring experience in Guides has improved to where creating holographic instructions is as easy as creating a PowerPoint, Kleiner added.
“I don’t have to hire an army of consultants to build this. I don’t have to have a bunch of people with computer science degrees. I can give it to the experts on the frontlines, and they can generate their own content to train other folks or share,” he said.
Toyota found immediate value in Guides as a training tool, Kleiner said. Instead of working one-on-one with trainees, Toyota trainers can let trainees work independently and supervise multiple trainees at the same time – increasing their efficiency many times over.
During a pilot project at their San Antonio plant, Toyota used HoloLens 2 and Guides to train employees on how to assemble a new version of the Toyota Tundra. The data showed it was a success, Kleiner said. Defects were cut in half. Depending on the individual, training time fell between 20% and 50%, he said.
Partners in mixed reality
While HoloLens 2 devices have helped define what’s possible in the industrial metaverse, Microsoft is a platform company, Taylor said. That’s why Microsoft is committed to making its mixed reality software available wherever its customers are – whether that’s on a HoloLens 2 or another company’s device. And while Dynamics 365 Mixed Reality Apps provide enterprise-grade software so customers can get to work immediately, Microsoft has also built a Mixed Reality partner network of ISVs who can extend solutions to meet unique needs in different industries, from construction and education to healthcare and pharmaceutical.
The U.S. Department of Labor’s Occupational Safety and Health Administration (OSHA) turned to one of these partners to improve worker safety during trench excavation. On construction sites, trenching and excavation are risky but essential jobs. One cubic yard of soil can weigh as much as a car, according to OSHA, and that delicate balance with gravity can be upset in an instant. That is why trench collapses are the leading cause of death in the construction industry.
To help mitigate this danger, OSHA instructors are using a custom mixed reality app to give trench safety inspectors immersive, hands-on hazard training without real-world risk. The app, created by the software studio Clirio, consists of six different scenes using realistic and immersive graphics to show variables like ground conditions, hazards, safety equipment and best practices for mitigating risk. Sophisticated sound design and animation add to the lifelike experience.
“Whenever you teach somebody something, it’s one thing to tell them, it’s one thing to show them, and it’s another thing to let them do it,” said Anthony Towey, Director of the OSHA Training Institute. “We can actually go into a trenching and excavation environment that’s as close as they’re going to get in the field.”
Thousands of construction contractors could use this training, he added. “And now they’ll be able to do it in a safe environment, where they can practice and make mistakes.”
Hokkaido Electric Power Company is using HoloLens 2, Dynamics 365 Remote Assist and a custom app to help staff inspect critical equipment at a thermal power plant. On every inspection patrol, workers navigate a vast labyrinth of boilers, turbines and generators and examine thousands of pieces of equipment for often subtle changes that can help them avoid larger problems, said Takaharu Umemoto, who works in the company’s Information Technology Section, Thermal Power Department. That requires extensive know-how and experience, he said. It would often take new hires one year of shadowing experienced technicians before they could handle everything.
Today, new hires get up to speed much quicker, he said, and the app has improved the efficiency of patrol inspections. The experience has made the company enthusiastic supporters of mixed reality technology, Umemoto said.
“I had an image of (mixed reality) as a technology for games, but it was a revelation to find that it can be used as an intuitive and easy-to-understand solution from the perspective of transferring patrol inspection skills,” Umemoto said.
‘Everyone has a voice now’
As early customer pilots with HoloLens 2 have led to larger deployments, Microsoft has invested heavily in making the cloud-connected technology easier to manage at scale. That means IT departments can manage a HoloLens 2 headset just like any other laptop, phone or tablet, Evans said.
Those improvements helped Toyota move HoloLens 2 from the lab to the enterprise seamlessly, Kleiner said. “Usually when we leave the lab, there’s a huge learning curve: how do we maintain and operate these at the enterprise level?” Kleiner said. “All that was avoided because when we took the device to the IT department they said ‘Oh, this is just another Windows machine. We know how to handle it.’”
Microsoft will continue to work with customers to solve unique and tough challenges, Evans noted.
Customers like the U.S. Army are helping improve both software and hardware, while others are helping drive industry-specific improvements that may eventually have broader application, Evans said. It’s akin to how technologies like ceramic brakes and variable valve timing first appeared in Formula 1 but ultimately went from the racetrack to everyday streets.
“The military program has its own set of requirements that are tuned to the needs of the soldier. So, it’s helping push the whole platform forward. It’s great to have early adopters that are driving requirements because you end up with this trickle-down effect,” Evans said.
One thing Evans has said Microsoft hears from customers is that, unlike consumers who expect a constant crop of new gadgets, businesses don’t want to have to replace their devices every two years. That causes too much churn. “No one wants to be obsoleted for 10% better capabilities. They don’t need a successor yet, but they want to know it will be there at the right time,” he said.
Evans said Microsoft is pushing forward on all core hardware technologies: display, tracking, sensors, battery life. “We’re just looking for the right design point to make it a meaningful update. They want a successor device that’s going to enable an even higher return on investment,” Evans said.
As the HoloLens hardware and software continue to evolve, Toyota’s Kleiner expects Guides to remain “the killer mixed reality app” for frontline workers. One day, it could be like Word, available on any device. But for now, Toyota will keep rolling out HoloLens 2 headsets across the company, giving frontline workers the tools to work and collaborate in new ways.
“We now have a device we can deploy to every person,” Kleiner said. “It’s easy to maintain, and it allows our workers to participate in the larger conversation, regardless of rank or team structure. Everyone has a voice now.”
Related Links:
- Read more: Holographic Microsoft Teams features empower hybrid collaboration in the industrial metaverse
- Learn more: HoloLens 2
- Read more: The Occupational Safety and Health Administration uses mixed reality to improve worker safety during trench excavation
- Read more: New possibilities for MR: HEPCO’s DX, stemming from the development of a patrol inspection application for thermal power stations
Top image: Desktop collaborators can annotate in 3D space and augment what a frontline worker wearing a HoloLens 2 is seeing with the latest update to Dynamics 365 Guides. (Image courtesy of Microsoft)
Jake Siegel writes about Microsoft research and innovation.