“See your body like never before – from the inside out.”
This was how the BBC described Your Body Uncovered, a television programme that uses Microsoft HoloLens 2 to help people understand their medical conditions and potential treatments by allowing them to see a digital replica of the inside of their bodies.
With the patients and their consultant each wearing the mixed reality headset, they can both see the same 3D images at the same time. When the consultant rotates, expands, shrinks or moves the images using hand gestures, the patient will see those changes, too.
This allows patients to embark on a unique guided tour of their medical issue, giving them a detailed yet easy-to-understand view of what’s happening inside their body.
There were six episodes in series one, each lasting around an hour, and they focused on conditions including cancer, COVID-19, stroke and gallstones.
To bring those to life in front of people, the programme’s producers, Remarkable TV, reached out to Dr Philip Pratt, a former Senior Research Fellow at Imperial College London, and Chief Scientific Officer at Medical iSight. The company uses HoloLens 2 to create 3D visualisations of the human body to allow surgeons to “look inside” patients and visualise their blood vessels, bones and internal organs.
“When the producers came to us they wanted to use virtual reality,” Philip says. “We told them ‘You could do that, but have you tried this?’ and gave them a HoloLens. As soon as they put it on, they were blown away. There was no discussion about which device to use after that, it had to be HoloLens.”
Patrick Furlong, executive producer of the show, loved what HoloLens could do.
“At the beginning we looked at virtual reality but that wouldn’t work because it’s hard to translate what people are seeing through the headsets to people watching on TV at home,” he says. “We wanted to show patients interacting with their body and experiencing it. We needed that interaction.”
“Philip showed us a digital replica of a skeleton using HoloLens and we all thought it was amazing. We realised we could use HoloLens to film everything we needed to, we could even flip the visor up and see patients’ faces and emotion.”
Furlong loved the HoloLens 2, but he also needed the BBC’s commissioning editor to agree to let them use it.
“We had a meeting with the BBC’s commissioning editor. We put a HoloLens on him and showed him a human heart beating in mixed reality. The editor had never seen anything like it. It was so powerful, and we saw the same reaction from the patients, too.”
Furlong and his team then approached 100 consultants and asked them to suggest patients who might be interested in appearing on the programme. They ended up with a mix of people – some had been told they needed surgery and didn’t understand what that meant, so they wanted to know more. Some weren’t sure if they wanted to undergo a procedure.
The patients were in control of their treatment at all times. They made all the decisions in partnership with their consultant.
During filming, the patient, their consultant and show hosts Kate Garraway and Dr Guddi Singh all wore HoloLens headsets and discussed the condition that had been replicated in front of them using mixed reality.
For some, looking at a replica of their own body and seeing how their condition affects it was too much to take. Tears were common, but it was always swiftly followed by relief and gratitude that they finally understood what their body was doing.
“That was an incredible moment,” Furlong adds. “The technology faded into the background and you’re looking at your own body, while you’re being told the truth by a consultant. It’s real, and that’s why it’s emotional, too.”
“Where some people think of mixed reality, they question whether the digital content can connect with people on an emotional level but this really did. It shows that technology doesn’t have to be cold and clinical, it can have a human application, too. It shows the power of technology on a very human level.”
The technology side of Your Body Uncovered was overseen by Pratt. He developed the experience that the participants could see through their HoloLens. He was also on set with Remarkable TV’s crew to make sure everything ran smoothly while they were using the headsets.
The consultants were very busy and took time out of their day job to film the show. That meant that there was no time to train them on using HoloLens. Luckily, the headset is easy to use – “I spent five minutes explaining to the consultants how HoloLens works, they put it on and away they went,” Pratt says.
To make sure the three headsets all recognised the same digital replica at the same time, Pratt used spatial anchors. Put simply, these are virtual “landmarks” in the room, which the HoloLens devices recognise and agree on where they are located and what they look like. HoloLens enabled Pratt to decide where to place these anchors quickly and easily. He could also save those anchors to use later, so he wouldn’t have to make the HoloLens devices learn the layout of the room every day.
The consultants could move and change the digital replicas using simple voice commands such as “next”, or hand gestures such as pinching and pulling to stretch the image.
However, the team also had to make sure that the people watching the show on TV could see the same things as the people using the HoloLens. When they moved or changed the digital replica in the room, the same movements and changes needed to happen for the people at home, too.
That was more challenging because the producers wanted the studio cameras to move – creating their own view of the digital replica that wasn’t being seen by anyone wearing a HoloLens.
Pratt came up with a unique solution.
“I attached a HoloLens 2 to the camera, so they moved together while looking at the same space,” he says. “Then we added in the digital replica in post-production.”
Your Body Uncovered was a huge success, and Furlong believes that was partly thanks to using HoloLens to give people a view of the human body they had never seen before.
“We couldn’t have made this programme without HoloLens 2,” he says. “There was no other technology that could have done it. It was such an immersive experience. People were deconstructing their own body and peering inside to see their organs. They were understanding what was happening to them in a way they had never experienced before.”
Pratt agrees the show was groundbreaking.
“If someone had an issue with their spine, a consultant would usually show them a generic model of a spine or draw something on a pad of paper. But now they could see their own body in fantastic detail,” he says. “I don’t think there has ever been a TV programme quite like it.”
