She wanted to take photos. He wanted to understand movies in English. Here’s how AI helped them both.

The last time she could see her family, her animals or her beloved flowers was when she was 11. Monique Van den Abbeel, born with congenital glaucoma that resulted in optic nerve damage, was blind in her right eye by age 4 and lost sight in her left eye seven years later. Several operations over the years had not changed anything for the girl from Bruges, Belgium.

The visual memories of her favorite people and things remained with her, but they were starting to fade. Now 43, and as spunky, defiant and independent as she was as a child, she yearned to use her smartphone by herself to photograph all those things that gave – and still give – her joy as a way “to help me trigger my memories, and hold them all,” she says.

In the town of Lokeren, about an hour’s drive from Bruges, a feisty 17-year-old boy likes swimming, listening to audio books and watching TV and streaming movies and programs with his family on the weekends when he is home from school.

In Belgium, many of the streamed movies and TV shows are in Dutch English, with Dutch subtitles. While Wouter De Brandt speaks Dutch, he has difficulty reading it because of Cerebral Visual Impairment, a result of a brain injury before and during his birth. When he tries to read, he gets nauseous, dizzy and disoriented. He wanted a way to hear the movies in Dutch that his family was watching.

Van den Abbeel and De Brandt don’t know each other, although Belgians like to say almost everyone knows everyone in the small country that is about 90 miles wide and 175 miles long.

But their hopes – Van den Abbeel’s to be able to take photographs, and De Brandt’s to understand the dialogue as his family watched movies – would be realized by the same small team of people who not only understood their frustrations but were determined to help them.

Among them was Katrien De Graeve, a largely self-taught programmer and web developer who is comfortable with technology, but not necessarily with attention. Born in France, De Graeve settled in Belgium, working for companies creating e-commerce sites and web applications.

In 2008, she was hired by Microsoft in Belgium, where for the last 11 years she has had a variety of roles, from developer evangelist to now as an Internet of Things specialist with the Azure Global Black Belt team of cloud architects that works with customers.

A year ago, De Graeve was asked to be one of eight participants on a new show, “Team Scheire,” a Belgian TV program similar to the BBC documentary series “The Big Life Fix,” where inventors create tech solutions for people who have special needs. (Featured in the BBC series was work by Microsoft researcher Haiyan Zhang, who developed a watch that can temporarily short-circuit the hand tremors of graphic designer Emma Lawton, who has Parkinson’s disease.)

De Graeve was on board, although not entirely at ease with the prospects of the TV show filming her brainstorming sessions with Microsoft colleagues, her meetings with Van den Abbeel and De Brandt as the projects progressed and her own late-night ramblings as she worked through problems by herself, often on weekends. It was not in her “comfort zone” – in part why she agreed to do it.

“I never thought about anything like this happening,” she says.

♦♦♦♦♦

Of the hundreds of requests for help that flooded into “Team Scheire” from the public ahead of the show’s first season last year, the host, technology and science journalist Lieven Scheire, and his team had to choose 16 people, with each inventor tasked with figuring out ways to help two people each.

Of all the problems, Van den Abbeel’s was “our moonshot case,” Scheire says.

He remembers an early session when the show’s staff met to talk about it. “We have a person who is blind who wants to take her own photographs without the help of anyone else,” he says. “Everybody looked at each other and thought like, ‘OK, we understand the need, but we don’t even see the beginning of the solution. How do you help a person who is blind to take photographs?’”

With the help of her colleagues, De Graeve thought there was a way. She also was resolute about helping De Brandt. In him, she saw a young man who wanted to not only enjoy entertainment time with his family, but also be able to talk about the latest shows and movies with friends at school, part of the currency of teen chatter.

♦♦♦♦♦

You know instantly – instantly – when talking to Monique Van den Abbeel that she is gregarious and spirited, with her deep laugh and excitement in her voice as she talks. She is a social media diva, with a presence on Instagram, Facebook and YouTube. She has her own website which she started in part to promote her 2016 autobiography, “Like to See! Blind Mama, Strong Woman.” She is the single mother of a 17-year-old son and hopes to be the first person in Belgium to have a guide horse (yes, that’s right, a guide horse).

Her son, Robin, is asked: Do any of your friends’ moms do as much online as your mom? “No! None of them!” he says, marveling at her energy and persistence, her drive to stay connected to the world around her. Van den Abbeel gives public lectures about living as a blind person and likes to entertain through shows that feature her comedy, singing and dancing.

Van den Abbeel has been taking photos of Robin all his life, figuring out how and where to photograph him based on his physical proximity and his voice. She has continued to take photos using a smartphone in much the same way. Robin helps his mom choose the photos that turn out best, and then posts them online for her.

But she wanted to do more than snap a shutter. She wanted to really know what she was capturing, and if it was in focus, or centered. And she knew she would have even more to photograph when a miniature horse named Dinky, about 2 feet high, becomes her guide animal in the next year.

Guide horses are unusual, but not unheard of, and Van den Abbeel already has had several meetings with Dinky, who is being trained now to work with her.

To help Van den Abbeel, De Graeve reached out to the Microsoft team behind the smartphone app Seeing AI, released in 2017 and designed to help people who are blind or have low vision by describing people, objects and text around them.

“Together with Saqib Shaikh, one of the people behind Seeing AI, we got talking about our ideas for Monique and how they saw an opportunity to leverage the groundwork of Seeing AI to create a special ‘for Monique’ prototype,” De Graeve wrote in a LinkedIn article. “Without this ground layer nothing would have been possible!”

Additional assistance came from De Graeve’s colleagues who took part in a Microsoft hackathon on the Redmond, Washington campus in April 2018. Among them was Wesley Backelant, a cloud solution architect with expertise in advanced analytics and AI.

“When I first heard about the project, I immediately felt that AI would be an important building block if we wanted to change people’s lives,” Backelant says. “Knowing that what we build would really impact the day-to-day life” of Van den Abbeel “immediately convinced me to join the team.”

What the team built incorporated additional machine learning so the camera can do real-time, offline recognition of 1,500 objects – even distinguishing vehicles such as “car” or “mini-van,” and then speak what those objects are to help Van den Abbeel choose her own photographs later on.

Also, when Van den Abbeel takes horizontal photos, if the shot isn’t straight, the phone will vibrate to let her know that, and it will alert her if someone’s eyes were closed in a photo so she can re-take it if she wants. These solutions, created for Van den Abbeel, are not publicly available yet.

Using a machine learning model, the team also added to Seeing AI’s facial recognition features to provide real-time recognition of Dinky.

On a day of testing the app, as Monique aimed her smartphone camera toward Dinky, she heard: “Dinky is to the left” in the frame.

Another sound, already incorporated into Seeing AI, lets her know if there is enough light in the frame to take the photo.

“My eyes will never see again, but it’s like this has given me a new way to see,” Van den Abbeel says.

♦♦♦♦♦

Wouter De Brandt is the son of two electrical engineers. Like his parents, he’s smart and he loves to learn. But Cerebral Visual Impairment makes reading letters and words literally painful. It’s a little easier for him with numbers, explains his mother, Sofie Huys.

With numbers, “You only have 1 to 9, and it’s no big deal if you have to read 1,2,3 separately or you see immediately 123,” she says. However if you have to read each character, one at a time, “it takes a lot of time and effort for longer words, and once you’re at the end, you forgot how it started.”

His limited range of vision also makes it a challenge for him to ride a bike, or go for a walk on his own.

During the week, De Brandt attends a school far from his family’s home. He comes back on weekends, when Huys, dad Pascal De Brandt and brother Arno enjoy watching those mainly-in-English movies and TV shows with Dutch subtitles. A staple of their entertainment diet has been “The Big Bang Theory,” a much-loved, long-running American comedy series about young physicists and their friends.

But De Brandt doesn’t know English and he can’t read the Dutch subtitles because of his disability. When they heard about “Team Scheire,” his parents immediately applied “as we are constantly looking for tools that can make Wouter’s life easier,” Huys says.

De Graeve incorporated AI by combining optical character recognition for the Dutch subtitles and text-to-speech to speak the subtitles. De Brandt uses a Surface laptop equipped with this solution and wears a bone-conducting headset (versus an acoustic headset) so he can hear the dialogue, without missing the music and sound effects emanating from the screen.

The teen was less than impressed with an early prototype that didn’t quite do the trick. He shrugged it off, much in the way teenagers often say they don’t really care when they really do. “He was hopeful but realized it wouldn’t be easy” to come up with a solution, “and to protect himself from disappointment, he did set his expectations rather low,” says Huys.

For De Graeve, it was frustrating as well. “I was pretty stressed, and I was not sure how to solve it,” she says, not wanting to let down the teen.

After months of work, including help from colleagues, De Graeve developed a more polished app for De Brandt (the software for the app is published as sample code, available to anyone, on GitHub).

De Brandt “was surprised and happy,” his mother says. “It worked much better than he expected it to.”

He spent a lot of time catching up on episodes of “The Big Bang Theory.” His take? Meh.

“We are all fond of ‘The Big Bang Theory,’ so we assumed that Wouter would like this too,” Huys says. “But unfortunately, it’s not his cup of tea. However, at least now he knows he doesn’t like it, and before he thought he was missing a lot by not being able to understand what it was about.”

The family “doesn’t feel guilty anymore watching an English film, because now he has the choice” of whether he wants to watch it or not, she says. “He does not join always, but he can decide for himself now, where before there was no choice.”

“Now he can talk with his friends at school, and say, ‘Yeah, I watched that.’ ‘No, I don’t like that.’ Or ‘Yes, that was good,’” says De Graeve. “Before, he felt when they started talking about a movie or something, he said, ‘No, I didn’t watch it, I don’t know what you’re talking about,’ so he felt left out. But now he just – he just wants to be normal like everyone else, and I think I helped him with some small step in that, so that is really special.”

♦♦♦♦♦

Nick Trogh, a Microsoft software engineer in Belgium, was among the team members who worked on both De Brandt and Van den Abbeel’s projects.

He developed the object detection functionality of the app for Van den Abbeel, which “allowed her in real-time to hear which objects were in the camera frame, and where they were located,” such as “top left,” or “center, bottom right.”

Because he has been working with AI and computer vision technology, he also provided insights to De Graeve “on how to optimize the process of recognizing subtitles on a TV set” for De Brandt.

“It was truly an honor to be able to participate in these projects, knowing that a few lines of code can make such a big difference in a person’s life,” Trogh says.

The first season of “Team Scheire” is over, but the work for De Graeve isn’t. She’s following up with De Brandt, refining his app so it’s easier to use at movie theaters. And she and Van den Abbeel have become good friends and sometimes go to the theater together. “I’m so inspired by her,” De Graeve says.

The technical expert, the one comfortable with technology, but not necessarily with attention, estimated a lot of things right with both projects. What she underestimated, she says, is the impact both De Brandt and Van den Abbeel had on her life.

“It has been an evolution as well for me,” De Graeve says. “I still get emotional when I start thinking about both of them. I really want them to feel part of a community, and not feel left out.”

Top image: Monique Van den Abbeel practices taking photos using her smartphone.