Medical imaging, paramedic training and immersive learning in the era of mixed reality

HoloLens

Medical imaging, paramedic training and immersive learning in the era of mixed reality

Ten years ago, the concept of medical students’ collectively analysing magnetic resonance imaging (MRI) and computed tomography (CT) scans in a three dimensional and interactive environment was a distant dream.

So too was the idea of an immersive digital learning space where real-world events too dangerous, too difficult, or too dynamic to re-enact, could be safely virtualised and practised by paramedicine, policing and nursing undergraduates.

Fast-forward to 2017, and the old adage of ‘practice makes perfect’ is being applied to areas of education where ‘practising’ was once impossible; where students could have only ever been theoretically trained in situations that are in actuality, sometimes a matter of life or death.

The introduction of augmented and mixed reality into some of the country’s leading universities is having a profound impact when it comes to changing the way students learn in scenarios such as these.

It’s providing a scalable, efficient solution to teach critical skills, while also altering behaviours and cognitive responses; more adequately preparing students to excel in their respective fields.

The stories coming from Western Sydney University (WesternU) and the University of Sydney are nothing short of inspiring.

WesternU, where Mixed Reality is the unlikely hero

The Digital Strategy and Innovation (DSI) team at WesternU have targeted mixed reality along with the Internet of Things (IoT) as high impact technologies that will support learning and research more broadly.

This year, WesternU is delivering a 360 virtual reality (VR) space for its Paramedicine course and working with the Social Sciences disciplines to expand the application of VR, while also exploring Augmented Reality (AR) to support research.

The University’s Deputy Chief Information & Digital Officer and Head of DSI, Sarah Chaloner says, “Augmented Reality using HoloLens technology presents unique opportunities to improve people’s quality of life – for example those coping with dementia or a brain injury following a stroke could be supported by augmented instructions overlaid on the real world once headsets become more lightweight and less cumbersome.”

Her team are developing standards for VR and AR to broaden the opportunities to other disciplines and make mixed reality part of business as usual for education and research.

Dion Strasiotto, Innovation Specialist in the DSI team believes HoloLens will engage students more than ever through its immersive nature, demonstrating the technology at the University’s recent ‘May the Fourth’ event as well as its upcoming Professional Staff Conference, where staff will be able to walk around new buildings and facilities without leaving the actual venue.

“Mixed reality technology is not constrained to the world of gaming, enormous fun though that is.

At WesternU, we see the boundless possibilities to make real impact and help ordinary people to do extraordinary things,” says Sarah.

Simulating intensive real-world training

For degrees such as forensic science and policing, where practical training scenarios can cost thousands of dollars to replicate, Microsoft HoloLens will allow many students to undertake such training without the associated costs or possibility of something going wrong.

In its most literal form, the new, immersive space at WesternU will take the form of a cylinder that’s about six metres in diameter. It will soon be able to display holographic objects, with a natural depth of space, enabling students to enter environments where they can retain a high level of sensory or situational awareness.

Dr Paul Simpson, Director of Academic Program for Paramedicine explains that simulation is such an important part of preparing paramedicine students to function effectively in diverse and dynamic scenarios.

“Paramedics have to provide clinical care in very challenging situations and uncontrolled environments – on the side of a busy motorway in the dark, in a dimly-lit nightclub surrounded by crowds of people and noise. So in addition to excellent clinical capability, our students are required to develop the skills to maintain their focus and situational awareness in these circumstances.”

“The immersive simulation space gives us a great opportunity to provide a safe place for our students to practice and develop those critical skills.  This means our students will be especially well equipped to provide excellent care when they qualify.”

Embracing mixed reality in his school, Dr Brian Stout, Deputy Dean of Social Science and Psychology says, “We’re working closely with the DSI team to leverage this kind of technology to extend the learning possibilities in our disciplines.”

“A bit like Paramedicine, Social Work is practiced in different places in our community – in prisons, schools, hospitals so with a mixed reality setup here at the University, we will provide the opportunity for the region’s future Social Workers to develop the necessary skills to be effective in their roles”.

The University of Sydney takes aim for 2020

The wider goal of the University of Sydney has always been to create and sustain an environment where the brightest researchers and the most promising students, whatever their social or cultural background, can realise their potential.

With sights firmly set on 2020, the University of Sydney aims to triple investment in its research, and build a distinctive and different model of education.

The ICT Innovation team at the Sydney is playing a key role in assisting the University to realise these ambitions with technology-enabled classrooms.

“Universities are traditionally slower moving than other organisations, and we can’t always adopt emerging technologies as quickly as we’d like to, so it’s my job to map out a pathway to how we do that,” said Jim Cook, Innovation Lead, at the University.

Faculty of Medicine the first to roll-out HoloLens

Sydney Medical School first experimented with Microsoft HoloLens in 2016, and instantly recognised the possibilities of its application in clinical training.

In partnership with the Westmead Education and Conference Centre and Westmead Clinical School at Westmead Hospital, the University has turned its focus toward medical imaging, and overcoming the challenges associated with the lag time of visualising patient CT scans and MRIs.

“Our end-goal is to generate real-time data from scans and then stream it from HoloLens. We’re looking at how quickly we can convert an inter-operational ultrasound to 3D, get it through a HoloLens head-set, and work with our academics to understand the rendering side of it. In in about five weeks we’ve gone from a concept, to a working prototype,” This work is being spearheaded by Professor David Cook, the University’s Academic Director for new initiatives at Westmead.

For the University of Sydney, HoloLens has had a profound impact on teaching and research; becoming an incredibly powerful tool to interpret scans, and make more informed judgements on clinical outcomes.

As the Innovation team at the University explains, “Hospitals are busy, loud spaces, and you can’t always find a quiet consulting room to sit down and have a discussion. But if you can visualise someone’s X-rays, or someone’s 3D scans inside the headset, it’s actually an extremely efficient way of maintaining patient privacy – using great exactness – in the discussion of that case.”

The current timeframe between taking a scan to developing an image is around 40 minutes, however, as the team at the University of Sydney has discovered, it is possible to create a high fidelity tool in less than an hour.

“It starts to become quite viable to do that more often with real patients.”

Improving knowledge retention and understanding complex problems

The way the Innovation team see it, HoloLens can also be applied to improving student knowledge retention and understanding of complex problems across the University’s courses.

Phil Poronnik, a Professor of Biomedical Sciences gives the example of teaching students about proteins or how antibodies bind to their targets: “Students can quickly download the protein structures from existing databases and begin visualising them in mixed reality almost immediately. Tools like HoloLens allow students to discuss these structural interactions in detail as they can see what their peers are seeing.”

Using HoloLens also helps to explain topologies and scale in a way that is much better than previous methods, with students being able to step inside a neuron or stand on the surface of a protein in a holographic environment.

Although the clinical applications are a continuing process and are in their early days, the benefits for education are seen to be immediate.

With these types of applications now being realised, eight separate schools within the University have expressed interest and are keen to embed either virtual or mixed reality into their classrooms from 2018.

Jim concludes, “By 2020 you’ll see a paradigm in the classroom where, as headsets get lighter, as fields of view get bigger and as costs comes down, the natural progression will be to replace regular computing options with mixed reality like HoloLens.”

Dion goes one step further,

“I think anyone who doesn’t adopt a mixed reality approach to education at some point through the next six-to-eight years, will probably fall by the wayside.”

“This is the way people will be computing. This is the way our students and teachers want to be computing, and first hand, we’ve been able to witness just how much they’re able to innovate and grow with it. If that’s not compelling enough, I’m at a loss to what is.”