Mixed reality being trialled by Qantas engineers, delivering innovation in learning digitalisation

Qantas has an exceptional safety record; it’s something that its customers, who between them make 55 million trips each year, prize highly.

Keeping each and every one of them safe is Qantas’ number one priority -always has been, always will be. Now state-of-the art mixed reality technology is looking to take that to the next level.

While passengers are sitting in the lounge waiting to board, Qantas aircraft engineers perform an engine run as part of their maintenance duties. To do that they need to know how to safely start the engines across the fleet.

Training them to do that soaks up expensive and scarce flight simulator time, with engineers flown to the simulators for their initial training – which often happens overnight to manage simulator availability. When engineers complete regular refresher training, to revisit how to deal with high risk events such as an engine fire, it’s often a theory training session with a fellow engineer, simply because of the cost and complexity of getting them back to the simulator.

Working with the Microsoft Mixed Reality Partner Program experts from Altoura, Qantas is currently putting a Microsoft HoloLens2 solution through its paces, on a trial engineering training program and also exploring how the technology could be deployed more widely across its business.

The mixed reality simulation also offers something that other training sessions can’t; it allows people to experience a dangerous situation safely, preparing them for even the most unlikely eventuality – again, with passenger safety front and centre.

HoloLens2 debuts virtual cockpit

It’s a hot morning in Sydney and a Qantas engineer is about to carry out an engine run on a Qantas 737-800.

But the engineer’s not in an airplane, or a flight simulator, or even in Sydney. The engineer is in Darwin, and dons a HoloLens2 to replicate exactly what they would see if they were sitting in the cockpit of the plane on the tarmac in Sydney.

The engineer works through the engine run process, prompted by on screen cues displayed in the HoloLens. At the same time, the system is collecting data that creates a feedback loop for Qantas about what they are doing and how well they are doing it.

The proof of concept, which was developed with Altoura over a period of around five months, digitises a sample of steps within an engine ground run (EGR) test. The next phase will look to build out the full 45 step process, including responding to an engine fire. It’s a dangerous scenario, that’s difficult to test and train for in a realistic environment.

The HoloLens2 however lets Qantas engineers train for emergency situations safely so that passengers stay safe.

It’s a solution that can also bridge time and space – allowing engineers to be trained anywhere, anytime, for any plane.

Engine Ground Runs are a critical maintenance activity that follow strict standards and procedures for the safe operation of aircraft.  Qantas engineers receive an in-depth two-day initial training and four hours re-currency training every two years covering operation, and emergency and safety procedures.

One instructor delivers the training and assessment for one or two engineers per session.  Initial training consists of theory and practical assessment in a simulator.  Engineers spend approximately eight hours in a simulator with the session conducted overnight due to scarce simulator availability.

Recurrent training is carried out by an experienced engineer assessor on-the-job, but without the use of a live aircraft to replicate the emergency safety situations that would be possible in a full simulator.

In 2018 Qantas engineers undertook over 170 initial two-day courses and nearly 500 four-hour recurrent annual sessions. Training and assessment are regulated by the Civil Aviation Safety Authority (CASA).

The proof of concept HoloLens system uses mixed reality to build a digital twin of a 737 Flight Deck. The learner sits in either the Captain’s or First Officer’s chair to run through procedures and checklists.

According to Amanda El Bahou, Group Learning Technology Manager for Qantas, “The solution is not intended to replace an instructor or current physical assessment practices for Engine Ground Runs.

“The key purpose is to allow learners to undertake an engine ground run in a safe, simulated environment – demonstrating responses to high risk events.” And the system has been designed mindful of the need for Qantas to be able to manage the platform itself.

As El Bahou explains; “The airline industry is changing rapidly and the needs of the business are changing very, very frequently. So from a learning organisation’s perspective, we need to be responsive to how we deliver learning to our team members.” Initial feedback on the trial is proving the HoloLens is effective and meaningful for users and presents opportunity for value.

The technology also ensures simulators are used to maximum advantage.

That’s important because each simulator can cost tens of millions of dollars, is anchored to one fixed location and the logistics of getting people into them are complex not least because of rostering requirements and jostling priorities. If engineers – and longer term, even pilots – are able to use mixed reality to get familiar with a digital cockpit first, then the time they do spend in the simulator can be made far more effective.

Peter McDonnell, head of Qantas Group technology, said that the airline’s HoloLens 2 trial was further evidence of the extent to which innovation was core to Qantas’ DNA.

“I think HoloLens is really exciting. I had the opportunity to experience it the other day, and it takes that virtual reality concept one step further – that ability to for our engineers and crew to be able to experience (high risk situations) in a safe and more stable environment.”

He said that HoloLens gave Qantas personnel the opportunity to “touch, control and move and change. I think we will see much more application in probably a number of our different training areas, or even in technology.”

It’s another demonstration of Qantas’ deep commitment to using state of the art technology to empower all its 29,000 workers and ensure customers get the best and safest experience in the air.

Data infused learning

Jamie Fleming is CEO and Bharat Ahluwalia the Chief Technology Officer of Altoura which has worked alongside Microsoft to develop the proof of concept for Qantas. The firm specialises in developing solutions focussed on enterprise productivity with a particular concentration on environments that are dangerous, difficult to access, expensive – or still on the drawing board.

According to Ahluwalia; “There are a lot of frontline workers whose productivities are influenced by the environment they are in and typically it’s expensive and not scalable to train them inside those environments in real life. So we want to take those environments, digitise them so that we can optimise and scale up training for enterprises.

“Qantas is a perfect example.”

For Qantas, the HoloLens2 solution allows the airline to maximise its use of its expensive physical assets – like the flight simulator – and also get a head start on training for new aircraft.

According to Lawrence Crumpton, mixed reality solutions and partners lead for Microsoft South East Asia; “As they start to take new aircraft and new fleets, they can train and update training instantaneously as they get new procedures. They don’t have to wait until they get a prototype engine. They can begin that immediately. “

Just as importantly as accelerating the training, the system has been designed to enhance it by collecting previously unavailable data, so the system can assess how different users are performing, how accomplished they are at a particular task, and provide feedback to trainers about areas where they might need extra support.

Fleming explains that data collected during a training session can be presented to the trainer. “We can build heat maps based on the vision of where they were or tracking their hands in the environment. So, we can get really fine grain, very specific and sophisticated metrics on what people are doing inside these environments.”

This delivers much deeper insight about the competencies of key personnel and also helps enhance the training.

According to Crumpton; “For the first time we get really in-depth analytics about what the user is experiencing. We see where their eyes are, we see where their hands are. We even see when they look away from the environment to each other and we can understand how they collaborate and cooperate in real time.”

While it is too early to have any metrics on Qantas’ proof of concept, in situations where mixed reality training systems have been deployed in clinical healthcare settings they have been shown to achieve equivalent training outcomes to more traditional live training techniques. Where they exceed traditional training is in their ability to scale, and be used anywhere, anytime.

Amanda El Bahou says that the proof of concept is testing a number of assumptions relating to mixed reality digital learning solutions. It will test whether there is increased learner retention; faster speed to competency; consistency in delivery and assessment and increased learner engagement.

The technology also points to improved experiential learning for high risk scenarios unable to be tested on live aircraft, such as an engine fire.

It should also make refresher courses easier to schedule, reduce the overall cost of training, and provide the opportunity to discuss with CASA the current and future roles of mixed reality training.

“What we are testing in the proof of concept is whether mixed reality will sufficiently simulate an experience that is a reliable test of competency,” she says.

“Is the resource easy to access, user intuitive? Are movements and gestures frictionless, and is the experience comfortable – suitable for learning? What is the learner engagement with a digital flight deck compared with a physical environment?

“Also, can an instructor use the tool to assess high risk activity and responses which is not able to currently be assessed outside a simulator?” says El Bahou.

Mixed reality extends training horizons

In theory there is no reason why HoloLens2 could not be used also to help train pilots. At present many airlines use a paper printout of a cockpit which trainee pilots sit in front of and then go through the motions of flying a plane. This helps develop the muscle memory associated with completing different physical procedures in the cockpit, and can be performed before the trainee pilot is let loose in a simulator.

According to Jamie Fleming; “It’s almost like choreographed sequences. You memorise the sequence of spatially how you’re moving your hands and your body through the aircraft as you run through all these pre-flight checks.

“Mixed reality is very exciting because it’s using this spatial learning capacity that humans have to remember where their body goes in space, just like an athlete or a gymnast, and to build on that muscle memory in a spatial environment.”

As Crumpton explains; “You’re going to flip this switch, turn this dial, move this throttle and HoloLens 2 is able to track your hand in order to give you a physical movement that is really, really realistic and based on what you actually have to do in the cockpit.

“So it’s not going to be the first time you do this when you sit inside a multimillion-dollar simulator. You know where every switch is. You know the direction you’ve got to flip it, you know the direction the dial has to turn. “

Using Microsoft’s Mixed Reality Toolkit (MRTK) Altoura has been able to track in fine detail, trainees’ hand movements and, armed with that data, to provide immediate feedback about the way they flicked a switch or moved the throttle.

Eye trackers meanwhile monitor what the user is looking at, and for how long – again providing data that was previously not available that can be used to enhance training and ultimately boost safety in the air.

HoloLens2 might also help to turn hard landings into soft landings.

In a 737-800 the quality of the landing can be influenced by the height of the pilot’s head in relation to the dashboard. HoloLens2 can be calibrated to show when a pilot’s head is too low or too high, allowing them to adjust their seats during training – and then remember that setting when they are in the plane – to ensure that they are in the sweet spot to ensure a nice soft landing.

Fleming adds; “That’s something that has been very difficult to do in the past to really make sure that a person is sitting at the appropriate height in the cockpit. And now it’s a five second task that you can do in HoloLens2 in a way that you could have never have done it before using other types of devices or even in an actual simulator.”

Altoura has made extensive use of Microsoft technologies – HoloLens2 and MRTK being prime examples – along with Microsoft Azure as its underlying computational platform. Azure SQL handles all the system’s database requirements, Azure Storage is used extensively, and Altoura has also created an Azure web app that runs its administrative portal so that end users, including Qantas, can create their own training content and protocols.

The solution integrates with Azure Spatial Anchors that allows physical and digital assets to co-exist, which is critical in these sorts of mixed reality environments.

Being a global company with operations and talent all over the world, Azure’s international reach and scalability has ensured Altoura’s development teams have been able to communicate and collaborate where and when suits them best.

Increasingly, Azure’s cognitive services are also being explored; Ahluwalia explains; “Azure neural text-to-speech for realistic-sounding text-to-speech; so somebody can type in what they want a virtual customer to say, and then we convert it using neural text-to-speech. So that when you’re in the experience, you’re not just reading the content but you’re hearing the voice of the words your customers see, the same thing.”

Qantas sets the pace with mixed reality training

Whilst still a proof of concept, the metrics attached to the Qantas project are already well understood.

First, Qantas will be able to assess how quickly someone learned a process using mixed reality, and expose fine grained detail by heat mapping, for example, how long people took to complete a training module, or whether they were looking in the right places when running through it.

According to Fleming; “We can demonstrate how proficient this person is, how many times they did the training. We can validate that they actually completed the training. And we can even start a recording of that training, so we can play it back if there are any questions.”

There are also cost savings anticipated from engineers being able to be trained at their home base, rather than flown to a flight simulator.

The dream vision, says Amanda El Bahou is; “To be able to put our team members into environments well ahead of time, before they actually need to operate those environments. So if you think about if we’re bringing a new aircraft into service or we’re needing to implement a new engine procedure, we don’t want the first time for our team members to be operating or doing those procedures, to be in the live aircraft.

“For us, the opportunity of mixed reality is really about how can we provide experiences and learning for our team members, so that they can operate safely and they can deliver the right service well ahead of time. “

The logical progression for mixed reality technology is that for some applications it might one day offer a full-blown alternative to physical training platforms such as flight simulators.

Fleming is optimistic, but predicts a gradual evolution; “What you’ll see is that there will be less reliance on completing all of the current training within these simulators. A lot will be offloaded into a mixed reality experience, and all of the refreshers and a lot of the early pilot training, can be done in mixed reality. You’ll reserve the time in the simulators for things where you really actually do need to feel the physical feedback of the yoke in the simulation.”

Fleming adds; “Qantas is really leading the charge in terms of R&D and validating mixed reality. They’re talking about beginning the certification process with the various authorities that oversee all of that. So, they are taking a significant leadership stance in the industry.”