AI and cloud combine to transform opportunity for people living with a disability

Just over four million Australian children and adults live with a disability; around the world disability impacts more than a billion people.

Artificial intelligence in the form of machine learning and cognitive services backed by powerful cloud computing can be a game changer – providing tools, apps and solutions that help people living with disabilities with daily challenges, support their connections and communications, and boost their ability to find employment.

Microsoft’s Australian partner community, responding to the AI for Accessibility Challenge, demonstrated the sorts of innovative solution that the technology now supports.

Part of Microsoft’s global AI for Good initiative, the AI for Accessibility Challenge and pitch day, provides access to Microsoft Azure cloud computing resources, the opportunity to apply for grants, and to make use of Microsoft Cognitive Services and Machine Learning to develop accessible and intelligent AI solutions to support people with disabilities.

Participants in the Challenge have access to Microsoft’s Accessibility Toolkit and Accessibility 101 training.

Forty Australian partners responded to the inaugural Challenge and in early April eight companies took part in a pitch day held at the new Sydney Microsoft Technology Centre, showcasing ten separate and potentially game-changing ideas.

According to Mark Leigh, General Manager, One Commercial Partner, Accessibility Lead, Microsoft Australia; “This is the first time Microsoft has held this event anywhere in the world. It goes to the very heart of our mission – which is to empower every person and every organisation on the planet to achieve more.

“When it comes to empowering our customers one of our biggest assets is the partner ecosystem, made up local of 11,000 partners and 300,000 people. We have been genuinely delighted about the level of interest and commitment from our partners in addressing this challenge.”

Globally there are 1.3 billion people dealing with disability and seeking support with accessibility. Given the four million plus impacted in Australia, Deloitte has estimated that there is a market for digital assistive technology in Australia that will be worth almost $1.4 billion by 2020.

There are also significant opportunities for other sectors of the economy to benefit from making their products and services more accessible.

Dr Yasmine Gray, founder of Getaboutable.com, offers the example of the “Purple Dollar,” being her description for the disposable income of people with disability. She said that the purple dollar in global tourism was worth $200 billion – $10 billion of which was in Australia.

Operators which took the time to make their services accessible, and their culture welcoming would likely be rewarded by extremely loyal customers she said.

The Microsoft Australia AI for Accessibility Challenge focusses on accessible and intelligent AI solutions to support:

• Employment: To positively impact the employment rate for people with disabilities through more intelligent technology.
• Daily life: To increase affordable access to technology for people with disabilities.
• Communication & Connection: Harness AI to help improve the speed, accuracy, and convenience of communication for people with disabilities.

Leigh notes; “The innovative ideas, the commitment to this important issue – and the level of technical expertise that was on display at the pitch day was extraordinary.

“For the four million Australians with some form of disability or accessibility challenge, most of that is unseen. What the solutions showcased at the pitch day demonstrate is that AI powered by cloud has the potential to transform lives, to help children learn, help adults find work, promote wellness at every level of society, and for everyone to be empowered to be their best selves.”

Pitching Partners

Partners selected to pitch in the Challenge were:

AKQA – The AKQA Sensekit is designed to empower the one in ten people who suffer sensory impairment – perhaps blind or deaf, or with low vision, or who are hard of hearing. Intended to be developed as a kit comprising various sensors and haptic wearables, this allows users to develop solutions that respond to their unique requirements. Using sensory substitution AKQA’s technology takes data from sensors, then making use of Edge Computing and Microsoft Cognitive Services, transforms that into a signal sent to a discreet haptic wearable. Over time users learn how to interpret vibration patterns from the wearables, so that the brain starts to extract meaning from the vibration to be able to recognise a face or understand a spoken greeting, for example.

Data #3 – One in six Australians is affected by some form of hearing impairment, but Data #3 believes that the impact is even greater because if hearing people cannot learn from people who use sign language to communicate, they too are affected. Using Microsoft technologies Data #3 has built prototype solutions that translate gesture and sign language into text or voice. Using artificial intelligence and Microsoft Kinect the proposed solution would be able to interpret even fine finger signing, and parse and contextualise person-to-person communications.

LiveTiles – HiJo is intended to act as a ‘friend’ for a student in a classroom. Children would communicate their feelings via a desk mounted device called a MoodBox, which can also collect classroom information such as temperature and whether students are sitting or standing. That information is instantly analysed using AI and the insights made available to teachers, and through a portal for parents, in order to promote and protect student wellbeing. The technology can help identify patterns that might provide clues about triggers for certain emotions, and also monitor how long children have been sitting at a task – and encourage them to get up and move around to promote wellbeing. The MoodPad keypad can be replaced with braille or icons to maximise accessibility.

PwC – MoodPal is a digital friend powered by AI. Loneliness is a growing epidemic, which, left unchecked can spark depression. Depression and anxiety are estimated to affect more than 1 billion people worldwide. MoodPal is conceived as an intelligent and ethical assistant that people can talk to, use to record thoughts, collect photos or music, and keep track of eating and sleeping patterns. Armed with insights about how people respond to certain events MoodPal could nudge users in the right direction, prompting people to take time out to reflect on something, offer consolation to people feeling down, or organise calendars so that people are encouraged to take breaks which might reduce the risk of anxiety and depression.

Simplfi – An Australian based social enterprise focused on empowering the 20 per cent of the population with mobility, vision, hearing and other assistance needs Simplfi is working with Getaboutable.com a user-driven online site that helps identify accessible travel destinations. Leveraging crowdsourced recommendations and insights, Getaboutable is exploring the use of Azure infrastructure and cognitive services to develop a recommendation engine for users of the service.

Readify – SightMate takes images from a body-worn camera, feeds then through AI, and sends real time alerts to a haptic wearable that informs users whether a crossing is flashing red or there is a pothole ahead. Using object detection and image segmentation, and with plans to leverage 5g mobile networks, the solution is designed to provide insights to users, warning them when they are approaching a kerb or at risk of stepping off a zebra crossing. Intended to integrate with APIs on route planners and bus live tracking SightMate is conceived as a way to empower users allowing them more independence.

RecordPoint – Misplacing your keys is a universal frustration that’s amplified for disabled people. Using CCTVs, computer vision and machine learning RecordPoint is looking to develop a solution that can tag and track items around the home. When a user wants to know where their keys are, he or she can interrogate the system using natural language processing and the system will respond with details of the last sighting of the keys.

Simplfi – Getaboutable.com is a crowdsourced online travel site for people with accessibility challenges. Using AI the team at Simpli.fi are looking to close the loop, using a blog sentiment analyser, that interprets the opinions of people with a disability about the venue’s accessibility, infrastructure, communication and attitude. Using Microsoft’s text analytics API the team wants to score the sentiment and provide reports for the business owner so that they can take informed action. As the API works in multiple languages the solution is applicable worldwide.

Publicis Sapient – Focussing on supporting people with Multiple Sclerosis (MS), Publicis Sapient has conceived an idea that uses edge-based devices and AI to reduce the impact of temperature fluctuations which can dramatically increase MS symptoms. Using mobile machine learning and a data architecture called Affinity, developed by the company, the solution is intended to monitor temperature and send signals to wearable cooling devices that can be turned on to help influence the sympathetic nervous system and reduce the risk of MS symptoms being exacerbated by temperature changes.

LiveTiles – The wellness of employees is an increasingly important consideration for enterprises worldwide. LiveSmiles is a free intelligent solution designed to help companies optimise the wellness journeys of their people. Built around Office 365 SharePoint the solution uses AI to interpret how people are accessing wellness programs, and leverages bot technology to proactively make wellness recommendations to users. Initially focussed on the corporate market LiveSmiles plans to next focus on the needs of K-12 education and aged care.

Winning solutions

The judges (Peter Horsley, founder of the Remarkable inclusive technology accelerator championed by the Cerebral Palsy Alliance; Simon Sharwood, editorial director for IT News and CRN Australia; Phil Barlow, Microsoft Australia manager of the ISV technical evangelist and Azure cloud architect team, Anita Sood, ISV and Next Generation Partnerships Lead, Microsoft Australia; and David Masters, corporate affairs director, Microsoft Australia) collectively praised the calibre of the solutions that were pitched at the AI for Accessibility Challenge and the clear commitment to make a difference to the quality of people’s lives.

The judges selected Simplfi as the overall winner for its use of Azure Machine Learning and Microsoft cognitive services to develop a recommendation engine for users of the Getaboutable platform – a crowdsourced site featuring accessible travel destinations.

The other three partners named to the top four are LiveTiles with its HiJo system designed to use AI to promote student wellbeing; AKQA, for its Sense Kit that will empower disabled people to build their own solutions using sensors, haptic wearables and AI; and Publicis Sapient for its idea to support people with Multiple Sclerosis by developing an AI-infused solution that helps people manage their symptoms by controlling temperature.

As part of the global AI for Good initiative Microsoft has set aside $US25 million to support innovation through AI for Accessibility grants over the next five years.

Microsoft is also partnering with the Remarkable inclusive technology accelerator, which is working with startups using technologies such as AI, bots, virtual reality, augmented reality, drones and 3D printing to develop transformative assistive solutions.

Microsoft has committed to support two of the Remarkable accelerator projects – Bookbot which is using machine learning to help children with dyslexia learn how to read, and JobMatcher which is using AI to help bring together jobseekers with a disability and employers with roles that are suited to their skills.