When workers for Lockheed Martin began assembling the crew seats for a spacecraft designed to return astronauts to the moon and pave the way for human exploration to Mars, they had no need for paper instructions or tablet screens to work from.
Everything they needed to see — from animations of how pieces fit together to engineering drawings to torque values for tightening bolts — was visible in HoloLens 2 devices that they wore.
The mixed reality headsets left their hands free to manipulate hardware. Voice commands guided them through every step, with holographic instructions overlaid on the relevant parts of the four seats that will be installed inside the crew module of the Orion spacecraft, which Lockheed Martin is building to support NASA’s Artemis program to carry humans to the moon and beyond.
“They didn’t have to refer back to a computer screen or paper drawings during that entire activity,” said Shelley Peterson, Lockheed Martin principal investigator for augmented and mixed reality. “Out on the shop floor they can put on the HoloLens 2 device, power it up, and it has all the content that they need to figure out how to do that task overlaid right there on the structure.”
Building a spacecraft requires millions of tasks, each with zero room for error, from attaching electrical cables in the correct pathways to lubricating joints and precisely locating thousands of tiny devices that measure how the craft performs under stress.
Lockheed Martin, the prime contractor building Orion, has employed HoloLens 2 on a variety of assembly tasks for the spacecraft that will be used in NASA’s Artemis II mission, the first to carry a crew of astronauts aboard Orion.
For some jobs that require lots of precise measuring by hand — such as marking locations for hundreds of fasteners on Orion’s spacecraft adapter jettison fairings — technicians using holographic instructions have finished those repetitive tasks 90 percent faster. The mixed reality headsets have also all but eliminated assembly mistakes, Peterson said. Lockheed Martin has experienced zero errors or rework requests on tasks in which workers were assisted by HoloLens headsets, which the company first deployed at the end of 2017, she said.
“The fact that we haven’t had any errors across all of these activities is phenomenal,” said Peterson, who oversees the company’s mixed reality initiatives.
“Usually when we’re considering new technologies we’re asking if there’s improved quality, if it’s faster or if it’s less expensive, and most people say you can only get two out of the three because there are always tradeoffs. What we’re finding with the HoloLens 2 is that we can hit all three, which is pretty unique,” Peterson said.
At its Ignite conference on Sept. 22, Microsoft announced it is expanding the global availability of HoloLens 2, which is now available for purchase in Italy, Netherlands, Switzerland, Spain, Austria, Sweden, Finland, Norway, Denmark, Belgium, Portugal, Poland, Singapore, Hong Kong and Taiwan.
We are super inspired to see what people are doing in the wild with HoloLens, and we are very excited that we have enough supply worldwide to bring it to this next set of regions,” said Microsoft Technical Fellow Alex Kipman.
The company also announced the private preview of a new mixed reality service called Azure Object Anchors. It allows HoloLens devices to recognize an object in the real world and map relevant instructions or visuals onto it without requiring any expert knowledge or barcodes to line those things up.
If an automaker wants to help a car owner do a simple repair job at their house using mixed reality, for instance, it’s more helpful for the owner to put on a headset and see an arrow pointing right to the oil pan gasket that needs to be removed, rather than asking a novice car mechanic to locate the gasket among all the other engine parts and then start following instructions, Kipman said.
“It’s a very hard computer vision problem, but the ability to recognize an object in the real world and then graft and map holograms onto it adds a tremendous amount of value,” he said.
Since HoloLens 2 began shipping to customers last November, Microsoft has continued to see demand for the device and Azure mixed reality services, Kipman said, especially in a time when remote collaboration is essential to help limit the spread of COVID-19.
Microsoft Dynamics 365 Remote Assist, for instance, allows people in two different physical locations to collaborate and solve problems in a shared mixed reality environment. If a firstline worker assembling a spacecraft, airplane or truck finds an obstruction where the next piece of hardware needs be installed, an engineer in another room or city can see through the worker’s HoloLens 2 exactly what the problem is and offer advice on how to adjust.
Health care providers treating COVID-19 patients at the Imperial College Healthcare NHS Trust in England are using HoloLens 2, Remote Assist and Microsoft Teams to minimize potential exposure to the virus. One doctor wearing a mixed reality headset can treat the patient in person, while the device sends a secure live video feed to a computer screen in a neighboring room. That allows other members of the healthcare team to see everything that doctor sees while remaining at a safe distance.
Mercedes-Benz USA is using HoloLens 2 headsets to help service technicians at its dealerships more quickly and efficiently repair vehicles. Using Remote Assist, they can troubleshoot a problem in real time by easily tapping into the company’s vast network of remote specialists with deep expertise across various car lines.
“In some industries, we’ve seen companies become willing to take the leap much faster,” Kipman said. “In a socially distanced time where travel can be risky, it’s much easier to just teleport there and see things through the eyes of the operators. Before that sounded kind of cool, like something out of a science fiction novel, but now it’s really a necessity.”
Lockheed Martin officials said the ability to collaborate remotely using HoloLens 2 has helped employees maintain social distance during the pandemic. It helps workers avoid passing paper or tablets back and forth, and it also allows people to problem solve without having to get on a plane or look over someone’s shoulder.
“Travel is really difficult right now, and if we can avoid just one troubleshooting trip, that HoloLens is worth it,” said Colin Sipe, an Orion and Human Landing Systems senior manager for Lockheed Martin.
The company also began seeing significant time and cost savings when technicians no longer had to break from their tasks to look at manuals or type on a computer screen, which may require crawling through tight spaces or taking off clean room gear. For a critical job like assembling the heat shield that keeps astronauts safe as they re-enter Earth’s atmosphere, written instructions are so complex that they can add up to 30 pounds’ worth of paperwork, Sipe said.
“Typically, it turns out that about half of their time is spent reviewing drawings, reviewing steps and entering data into the computer, and only about half of their time is spent turning wrenches and putting parts together,” he said. “With HoloLens, we’ve been able to reduce the amount of time that they spend dealing with or processing that data overhead by about 90 percent.”
For instance, as part of critical verification processes, technicians often have to put down their tools and walk back to a computer workstation and type in information about the step they’ve just completed. With the HoloLens 2, they can use a simple voice command to have the headset take a picture or record a video — which can be used for validation, quality assurance or training purposes — without having to break their flow.
The people who build the spacecraft also enjoy using the HoloLens 2, which allows them to focus on the work they really like to do, Sipe said. Some have even delayed retirements to spend more time working on Orion.
“They don’t particularly like digging through technical documents to find out what kind of primer to put on this particular part. They like to build stuff and work with their hands, and with the HoloLens 2 they’re able to spend almost all day doing just that,” he said. “They really feel like they’re working on a spacecraft of the future.”
Top image: Technicians used HoloLens 2 devices to assemble the crew module seats, similar to these seats in a mockup of the Orion spacecraft, for the first Orion mission to carry astronauts into space. Image courtesy of NASA.
Related:
- Learn more: HoloLens 2
- Learn more: NASA and Lockheed Martin Orion spacecraft pages
- Learn more: NASA’s Artemis Mission
- Read more: The making of the HoloLens 2: How advanced AI built Microsoft’s vision for ubiquitous computing
- Read more: HoloLens 2 gives Microsoft the edge in next generation of computing
Jennifer Langston writes about Microsoft research and innovation. Follow her on Twitter.