MDA: Reimagining space missions with mixed reality

By Kevin Peesker, President, Microsoft Canada

Rapid innovation in advanced technology is proving that anything is possible – the sky is ‘literally’ no longer the limit. Mixed reality is pushing boundaries and enabling remote collaboration from anywhere on Earth and one day soon, space. MDA, an international space mission partner and space technology company, is pioneering the use of mixed reality to solve mission critical challenges, support on-orbit robotic operations and provide immersive, in-situation training for astronauts.

Based in Brampton, Ontario, MDA’s work focuses on three main areas: geointelligence, satellite systems, and robotics and space operations. The development and operation of Canadarm and Canadarm2 is a huge source of pride for Canada – it has earned a spot on the five-dollar bill! The robotic arm has supported space shuttle missions, deployed satellites, serviced the Hubble Space Telescope and played a major part in the assembly of the International Space Station. As part of its partnership with the Canadian Space Agency, MDA is now developing the Canadarm3, a next-generation AI-enabled robotic system destined for Lunar Gateway, a lunar-orbiting international space station.

Industry collaboration is fundamental in fueling innovation that meets the unique needs of space. From immersive visualization using the HoloLens 2 to cloud-powered technology using Azure Space, Microsoft is committed to helping its customers and partners achieve success in space missions by re-imagining solutions to some of the most challenging problems. By collaborating with Microsoft, MDA is unlocking the potential of mixed reality and cloud-computing capabilities both on Earth and in orbit.


One of the biggest challenges when developing space technology is simulating the on-orbit experience and incorporating the realities that astronauts face. This is where MDA’s Dynamic Robotic Emulation and Mixed Reality (DREAMR) Lab comes in. The DREAMR Lab is a zero-gravity robotics lab that integrates mixed reality, where engineers can test robotics to see what the actual performance of the hardware is going to be on-orbit.

“By using mixed reality, we’re able to incorporate context and visuals that you can’t normally experience here on Earth,” said Mike Hiltz, Department Head, Mission Operations at MDA. “While much of the testing is at the physical critical location, the robot-to-payload interface, Microsoft HoloLens 2 is able to augment the analysis and provide insights that you can’t get in the lab directly, including realistic perspectives from space.”

Leveraging the open-sourced Mixed Reality Toolkit, MDA has explored diverse ways that elevate their research and development capabilities. For example, using the HoloLens 2, when MDA engineers move the holographic arm, the real robotic arm moves with it. This reduces the need for multiple people; one engineer can stand next to the robot, give it a push and receive all of the pertinent information right in front of them. This is particularly important when it comes to visualizing information from sensors, such as force measurement and position tracking.

MDA is also using an immersive environment to test prototypes of how they plan and execute missions. Telemetry, or ‘data’, performs a significant role in the DREAMR Lab during development, as well as during the analysis phase, by visualizing all the variables including two robots, an emulated spacecraft, as well as the forces and moments at play. During orbits where communication is intermittent, ground controllers will be able to quickly reconstruct what happened in a “time-out” period by playing back the telemetry and looking for anomalies. Currently, this information is represented as numbers on a graph – by leveraging AI technology in conjunction with mixed reality, the HoloLens will bring this data to life in a very visual way by showing how it looks on the space station.

Training for On-Orbit Operations

MDA’s goal with the HoloLens 2 is to create an immersive environment that visualizes the space station and the incredible views that astronauts would see on-orbit to capture the ‘awe factor’.

“Training astronauts and ground controllers on how to use space robotics on Earth presented a unique opportunity,” said Kevin Nasimok, Operations Team Lead at MDA. “At the Canadian Space Agency, every astronaut that goes to the space station receives robotics training to understand how to operate Canadarm2. In reality, that often means they have to pluck a spacecraft out of space, which is a difficult task because it’s not fixed to the station and everything is in motion.”

90 percent of on-orbit operations are controlled from the ground and there is a need for highly skilled ground operators who are on-point when things don’t go as planned. MDA is using mixed reality to strip away the layers of a model and allow someone wearing a headset to stick their heads inside the robotics to visualize how the mechanisms are moving. “In the teaching process, we’re able to show them what’s going on inside. It’s very powerful because it gives them a better feel for how things are moving and responding,” said Nasimok. “In these lessons I get a lot of ‘Oh! That’s how it works!’ because it’s hard to visualize otherwise.”

Demonstrating the sheer scale of working in space is an important component of training, and mixed reality better prepares astronauts to operate the 17-meter-long robotic arm on a space station that is the size of a football field in length. Using Azure Spatial Anchors, MDA pre-positions all the assets so that any time the HoloLens is used in a specific room, everything is automatically in the right place. As well, when multiple users are in a single physical space it ensures that each user sees the same image, in the same location. If users are in two different physical sites, MDA uses two different Azure Spatial Anchors that provide reference frames so that a user’s position in the one room relative to the anchor is reflected accurately in the other physical room. Eventually, MDA envisions this evolving to Microsoft Mesh to connect and collaborate with greater depth and dimension.


What’s Next

As the next big leap, Canadarm3 is MDA’s most ambitious space exploration project yet. This robotic arm will be used aboard Gateway, a NASA-led deep space outpost that will orbit the Moon. This mission presents new challenges for MDA because the orbit has very intermittent communications. By using advanced AI-enabled sensors and systems, Canadarm3 will conduct operations without requiring oversight and monitoring by the ground controllers or on-board astronauts. With these extensive “time out” periods, mixed reality is expected to play a critical role in the reassembly, as well as providing ground controllers the ability to collaborate and operate the robot remotely.

By working together, Microsoft and MDA are discovering new insights, breaking down barriers and unleashing the potential of technology to change the world for the better, on the ground and in the stars.

For more information, please visit: Mixed Reality Deep Dive | Microsoft HoloLens

Related Posts