Mixed reality apps from Actimage help those with autism safely prepare for the real world

When the 12-year-old girl walked, she routinely looked down at her feet, not ahead. It was not unusual for her, or for others who have autism. Eye contact with the world around them can be a challenge.

But using a HoloLens mixed-reality headset and an application called Pop Balloons, the girl felt more comfortable looking ahead. The app rewarded her with points when she found and popped balloons with her finger, and the faster she got at popping them, the more points she scored. Most importantly, as she used the app, she became more aware of her surroundings, improving her motor and relational skills.

Pop Balloons is one of five applications created by Actimage as part of its Hol’Autisme suite to help those with autism improve their motor skills and learn social skills. It’s not the only work Actimage does. The company’s broad portfolio includes mobile apps, cloud services, data intelligence, project coaching, machine learning, user experience design, motion design and DevOps.

“Innovation today is really increasing the capabilities of people with disabilities, and Actimage is probably one of the best partners in mixed reality to help people with disabilities,” says Philippe Trotin, accessibility lead at Microsoft France, who has worked with Actimage to develop the programs.

Mixed reality merges the real and virtual worlds. Using a HoloLens means being able to visualize, hear and interact with virtual elements while you’re in the real world. You see what surrounds you in real life, but holograms are added to the view. In the case of Pop Balloons, you’re looking to pop them in the real world, but the balloons themselves are holograms. It is an effective – and safe – setting in which to explore and learn.

Three years ago, Actimage became one of the first companies to develop applications for HoloLens, starting with Pop Balloons. It came at a time when the government of France was working to improve the country’s health policies and social attitudes about autism, something previous administrations had not done.

“Actimage is leveraging digital technologies that are a great help for people with autism to develop and progress in their daily lives,” says Lucas Ninard, Actimage business project consultant. “We want to help provide them with tools and fill the gap between them and the rest of the population.”

Actimage’s Hol’Autisme suite is being tested in clinical studies in research and medical facilities, including the Cognacq-Jay Foundation and the Ellen Poidatz Foundation, which are trying to learn more about degrees of autism. One of Actimage’s new programs, JobNow, just started being tested by the government agency Pôle Emploi, which helps those who are unemployed find jobs and get financial assistance.

The application is meant to help those with autism get past a big hurdle in the hiring process: that first interview with a person from a company.  In France, employers with at least 20 employees must have workers with disabilities represent at least 6 percent of their total workforce, including full- and part-time employees.

JobNow uses a virtual coach to conduct a practice interview, asking the candidate questions about his or her background, experiences and goals. In real life, such an interview can be stressful for someone with autism, exponentially so than for those who do not have autism.

“One statistic that was a starting point for us with Job Now is that in France, 98 percent of people with autism are unemployed,” says Ninard.

Microsoft’s Trotin says that when most young people pursue their first job, “Typically, they will go to a company – they are going to interview with one, two, three, maybe 10 companies before being hired,” and they expect it to be a potentially lengthy process.

“It is much more difficult for a person with autism because they are not sure how to prepare for that first job interview,” he says. “Maybe they will go on to another job interview somewhere else. But then, if that doesn’t go well, they will stop looking for work, feeling their disability won’t help them succeed, and that they will never find a job.

“JobNow can provide an environment where they can enhance their potential of such meetings with HR people, to be well prepared, to reduce the level of stress, because they will have in front of them a hologram and not be judged by the hologram,” he says.

Click here to load media

After the meeting, the interview candidate is mailed a printout of the interview questions and answers for review with a real-life coach so that the candidate can better prepare for a real-life interview. There is also a summary of some key performance indicators based on the candidate’s answers, analyzed for factors such as the length of an answer, or the rate at which a candidate speaks. These tools can increase a job candidate’s chances of success, Trotin says.

Other applications in Actimage’s Hol’Autisme suite are geared to children and teens. In addition to Pop Balloons, there other programs: Smile, Across and EduFriend.

Pop Balloons is about improving motion control, but it also improves communication.

“In the app, there is a virtual guide, a little girl, who gives hints about which balloons to pop and where they’re supposed to pop,” Ninard says. “The game is about understanding simple cues given by someone and acting on them.”

Recognizing what others’ facial expressions mean can sometimes be difficult for those with autism, and the Smile app aims to help with that. It uses Azure Cognitive Services to identify and recognize a person’s facial expressions, with progress noted in a quiz asking the user to assign emotions to photos.

Crossing streets where there is traffic can be another stressful – and dangerous – situation for some with autism, especially if they are not focusing on where they are walking or paying attention to traffic or signal light changes. That’s where Across is meant to help.

“It’s about both hearing something and looking at something, because there is acute sensitivity to sounds and rapid movement for those with autism,” Trotin says.

“What is natural for somebody – saying OK, the light is red, the car is stopped, and I can cross – sometimes that’s too much information for a person with autism to analyze. The application is there to help a person learn to analyze that, to have extra information, to practice: The light is red, the car is stopped, you have no sound, so you can cross the road.”

The Hol’Autisme platform monitors user behavior anonymously during these play sessions, recording data such as reaction time, mobility stress and heartbeat. The goal is to see whether there are trends that can be used to predict and understand autism better, ultimately helping families and the medical community.

EduFriend is more for teens, and is centered around social interaction, Ninard says. “There is a list of questions they can answer using the HoloLens, with privacy. They’re the only ones to see the questions and the answers.”

Actimage, which has offices in France, Germany, Luxembourg, Switzerland and the United States, is a Microsoft Mixed Reality Partner Program (MRPP). As such, Actimage provides training for other customers and developers about HoloLens and mixed-reality applications.

With its knowledge and wide reach, “Actimage can help to change the life of people with disabilities, and especially autism,” says Trotin.

World Autism Awareness Day, designated by the United Nations, is  April 2 every year.

Top photo: Actimage’s Smile app helps people with autism recognize what others’ facial expressions mean. (Courtesy of Actimage)