Countless young people would love to make a career from gaming. But a clever Kiwi skilling organisation, Skills Consulting Group, is helping them gain a career through gaming instead. With Microsoft Dynamics 365 Guides on Microsoft HoloLens headsets, it’s supporting training provider ICE to give hands-on lessons using mixed reality modules that let apprentices practise and learn at the same time, in the kind of gamified environment they’re already at home with. Now even those who struggle with reading can level up as fast as their peers – a game-changer for industries across Aotearoa.
Collectively, billions of hours have been racked up by gamers across the globe playing the most famous plumbers in history – Mario and his brother, Luigi.
But actually apprenticing as a plumber and gasfitter is typically less exciting, involving a lot of theory, wasted materials and guesswork. Imagine only having encountered a gas pipe in a textbook, then suddenly being faced with a leaky, pressurised (and potentially dangerous) piece of equipment in front of a tutor who is marking your work. It doesn’t do wonders for your recall.
But what if you could apply the skills you’ve learned from gaming to the real thing?
James Meyer is Group ICT Manager at Skills Consulting Group, an organisation that oversees qualifications, standards and assessments for 21 trades across New Zealand. Always on the lookout for new ways to help training providers enhance the learning experience with technology, he couldn’t hide his enthusiasm when he tried out the Microsoft HoloLens mixed reality headsets.
“I’m trying to improve the learning experience without students having to do things over and over again. Being dyslexic myself, I’m also looking for ways to improve the teaching of theory. The beauty of HoloLens is you can show trainees how to do it the right way first time, by overlaying Dynamics 365 Guides onto the real thing. It teaches you the practical step-by-step process and the theory all in one go.”
Setting the stage for hands-on success
To make training more efficient, repeatable and effective, Skills Consulting Group is using Microsoft Dynamics 365 Guides on Microsoft HoloLens 2. Trainers can quickly and easily create, update and deliver content to multiple trainees at the same time, while tracking their progress to identify opportunities to optimise training.
James arranged for Microsoft specialist partner, Velrada, to demonstrate the technology at Auckland training provider Industry ICE for the tutors of ICE’s plumbing, drainlaying and gasfitting courses.
As with any new technology, it took a bit of fine-tuning to get things just right. External consultants were brought in to design tutorials for the headsets, but they weren’t familiar with the technology. Instead, Velrada’s HoloLens Consultant spent time on site to coach the tutors on how to get the most out of the headsets and design their own simple guides.
“That made all the difference – having someone who understands the technology take the time to sit down and teach them and find out exactly what they needed,” says James.
Lecturers learned how to load CAD drawings, photos, videos and hand-drawn guides into the HoloLens tool, creating specialised tutorials for each key part of the training. Velrada even designed a PowerApp that enabled tutors to time students performing set tasks – and better yet, their actions while wearing the headsets could be recorded, so they could watch everything back and learn from any mistakes, while having visible evidence of achieving each standard.
The whole training floor at ICE was set up for HoloLens tutorials and assessments, with each gas hob, workbench and pipe tagged with its own QR code linked to the appropriate tutorial. Portable roofing rigs and solid fuel fireplace rigs were also built. Each student could then calibrate the visual display to their own height, so little electronic arrows or guides flashed up in exactly the right place to show them where to look for the leak on the equipment and make adjustments. In the case of the roofing and solid fuel rigs, it showed them how to bend the galvanised sheet metal flashings or assemble tricky components with a little video or written or narrated instruction to tell them what they needed to do.
Once the tutors and students got up to speed, even more features were added, including quizzes that enabled students to compete against one another in fun challenges.
“There’s no fear anymore”
The results blow the old-school approach out of the water.
“It’s almost like having another tutor in the room. Students can take their own time to learn while I’m teaching the rest of the class,” says tutor Adam Worth.
“When I first tried it on, I couldn’t believe the technology was here to do this. It was just like in the movies. Every student who’s seen it is blown away – their eyes just light up. And best of all, it’s second nature to them, very similar to using a tablet in the way you can “pinch” the virtual screen to move it around and press virtual buttons to move to the next slide. You can even press buttons just by focusing your eyes on them. There’s no fear for them anymore, because it’s all digital.”
Using Dynamics 365 Guides, students can go through procedures one step at a time, as many times as they want. This removes restrictions to in-person training and gives both trainers and students new flexibility around when and how fast they learn.
Velrada’s Chief Innovation Officer, Dan Hookham, points out that safety has also been much improved.
“Students in practical subjects are not wired to sit down and digest a lot of theory. They want to get started and get their hands dirty, and mixed reality has allowed them to work with the real thing in a safe way, being shown which safety procedures they need to follow as they go.”
James is also delighted at the reduction in waste. Typically, training a student to bend and cut metal flashings to align with roof tiles and downpipes takes at least 10 pieces of sheet metal. With virtual guide lines showing apprentices exactly where to bend their piece of metal, they can now get it right first time.
“It enables them to build a muscle memory much faster,” he says. “It’s just like rote learning.”
The ICE team have timed students building a fireplace for the first time. With the HoloLens, it takes novices less than an hour and three quarters – without, it can take a full day. James talks proudly about a bricklaying class who managed to build a highly complex curved wall in a few hours, when it would have taken professionals several days. By programming the correct shape into the HoloLens, it mapped all the curves onto the site in the right place – staying green when students placed the bricks correctly, and turning red if they went over the line.
One of the most important benefits has been making learning easier for those who often get left behind. Adam says a significant number of students each year require reader-writers to help them through coursework, because of low literacy, dyslexia or even colour-blindness. Because the HoloLens can narrate each step out loud for wearers, or change the screen colour or font to make text easier to read for those with dyslexia, students are now able to keep up without the need for a reader-writer.
“It dramatically improves their self-esteem, because they can do the task on their own without help,” Adam says. “We’re also using PowerBI to monitor how long it takes students to perform tasks wearing their headsets, so we can see where individuals are struggling and provide more support.”
The future – remote tutors and virtual coffee-making
Lead course designer at ICE, Rob Hatchell, predicts huge growth in HoloLens-supported training and working within the next few years.
“We’re looking to licence HoloLens testing to other providers in future, and companies will be able to do staff inductions with them too,” he says. “And in the workplace, if you need to check on which fitting you need when you’re on site, you can give your boss a quick call, he can see the pipe through your HoloLens and tell you in an instant which one to use.”
ICE is also planning to set up a virtual bench like the one it uses in real life, assessing students on how many faults they can locate to achieve their final standards.
“If Covid or anything else means they can’t come in for assessments, we’ll be able to assess them from anywhere in future. Some of the apprentices have been set back years in their careers thanks to lockdowns, because if they haven’t passed certain stages they don’t progress to higher salaries. In future, because we can see exactly what they see through the HoloLens, we’ll be able to assess them remotely,” explains Adam.
“That could enable us to take on more students via online learning,” adds Rob. “Already, remote assist mode enables tutors to support students from another site.”
Emma Barrett, Public Sector Director at Microsoft New Zealand, says she’s excited to see what else the training sector can do with mixed reality. HoloLens headsets have also been used by nursing students at Southern Institute of Technology to practise treating Covid patients without needing actual patients.
“At a time when we’re seeing significant global skills shortages, the ability to train students and apprentices faster and provide richer learning opportunities is crucial. It’s been wonderful to see how innovative the New Zealand higher education sector has been with mixed reality, and as more and more institutions embrace it, the benefits to students and employers alike are set to be huge,” she says.
Skills Consulting Group has plans with Velrada to roll out another 25 or so headsets to training providers in the near future, including the upcoming HoloLens sunglasses, to enable more outdoor lessons. James envisions hospitality tutors using them to show students how to make the perfect coffee.
“Mixed reality just makes everything less of a struggle,” says Rob.