Microsoft’s Seeing AI app expands horizons for blind and low vision users

ANU law lecturer Cameron Roles is adamant; “Now is definitely, in my view, the most exciting time in human history to be blind.”

One of triplets, Roles was born three months early and the oxygen that saved his life caused his blindness. It did not however diminish Roles then or now, and continual advances in technology, particularly the latest accessibility features in platforms such as Office 365 and Windows 10, mean he is able to live a full and highly productive life.

Microsoft’s free Seeing AI app is opening up even more possibilities. It’s a pocket sized smartphone based narrator for the world around him that can be used when and where he wants. Seeing AI is an ongoing research project that brings together deep learning on the device and the cloud computing power of Microsoft Cognitive Services to deliver a useful, intelligent app.

It means he can, for example, point his smartphone at the pantry to find the peanut butter when the kids come home from school, and then use it to narrate their homework questions.

Seeing AI uses Microsoft AI and the camera on the iPhone to read documents, recognise people’s faces and provide a description of their appearance, identify a product using its barcode, and recognise and narrate images.

For the estimated 384,000 Australians [1] who are blind or have low vision Seeing AI is a “beautifully simple app that describes people, things and text,” according to Jenny Lay-Flurrie, Chief Accessibility Officer, Microsoft.

“We’ve providing computers with the intelligent capabilities to see, hear, talk and understand natural ways of communication. This has profound implications for enterprise customers but also allows us to develop tools that promote inclusivity and allow more people to benefit from digital innovation,” says Lay-Flurrie.

Microsoft’s work in AI means it now has solutions that can recognise speech with an error rate of just 5.9 percent – and images with an error rate of only 3.5 percent.

It also means that computers on a factory floor can distinguish between a fabricated part and a human arm, or that an autonomous vehicle can tell the difference between a bouncing ball and a toddler skipping across the street.

For Cameron Roles, who is also a director with Vision Australia, Seeing AI is “an example of Microsoft taking a leap forward and saying right, we’re going to try and step out and solve a problem using some of our research and our ability to innovate, and that’s exactly what they’ve done.”

Seeing AI can read documents, and make sense of structural elements such as headings, paragraphs and lists so that people can quickly navigate a document using voiceover.

It can identify a product based on its barcode and read the label. It can recognise images in other apps, even recognise people and provide a description of their appearance including gender, facial expression and other characteristics.

Roles believes that assistive technologies will play an important role for individuals, and society as a whole, empowering people to more easily access important content, while expanding the workforce to include a larger base of talented people.

It’s music to the ears of Vision Australia CEO Ron Hooton. “Our motto is ‘Blindness. Low Vision. Opportunity.’ These emerging assistive and inclusive technologies bring that concept to life for so many people who can now play a much fuller role in society.

“A smartphone app like Seeing AI is unobtrusive but effective. Blind and low vision users who might have struggled to read a menu when they are out at dinner, or select a product in the supermarket are now able to do that instantly and simply.”

Kenny Johar Singh is a Microsoft cloud solutions architect based in Melbourne. He originally wanted to be a doctor until a degenerative retinal condition changed his career plans.

But like Cameron Roles a lack of vision wasn’t going to limit his aspirations.

Singh says; “I took a pivot into computers because they were the mechanisms I was using to access information. As I was losing my sight computers were quite an empowering force, so I thought ‘let’s make a career out of this’.”

And he has – a very successful career.

At work Singh uses a range of assistive and Microsoft technologies to allow him equal access to information and insight as his fully sighted peers.

Meanwhile as an early adopter of Seeing AI Singh says the app; “Gives me access to the physical world.

“I use it a lot to identify objects products and to read documents if I get them in a physical printed format. And I’m experimenting with the person identification capability.”

It points to both exciting times and expanding horizons for blind and low vision members of society.