Haleon harnesses Microsoft’s Seeing AI technology to make health product information more accessible for people who are blind or have low vision

For Marc Powell, a seemingly simple trip to the store to buy medication or other consumer health products is often an onerous, complicated undertaking.

Powell, who is blind, usually must ask a store employee to help find what he needs and then read the information on the packaging to him, which can feel intrusive. At home he’ll ask his partner, who has partial sight, for help, but the print is sometimes too small for her to read. In a pinch, he’ll scour the internet or wait until a family member stops by and can read aloud to him the information he needs.

“I’m incredibly reliant on somebody else to let me know about the product — what I’m meant to do with it, how many tablets I can take or what it contains,” says Powell, the accessibility innovation lead for the U.K.-based Royal National Institute of Blind People (RNIB), who lives in Cambridge, England. “It’s crazy to say that, isn’t it?”

A new collaboration between Microsoft and global consumer health company Haleon aims to make health product information more accessible for people who are blind or have partial vision, or individuals with low literacy. The companies worked together to expand the functionality of Microsoft’s Seeing AI app to provide detailed audio information for more than 1,500 Haleon products in the U.S. and the U.K., including brands such as Centrum, Sensodyne and Emergen-C. The initiative was launched in recognition of World Sight Day Oct. 13.

Using Seeing AI, users can now scan the bar code on Haleon products and hear the same information that is provided on the packaging, such as the product name, usage instructions and ingredients. When users point a phone at a Haleon product, the Seeing AI app guides them to the bar code with a series of intensifying beeps. After scanning the code, users can navigate between sections to get the specific information they want, a feature that has been well-received by partially sighted people who tried out the functionality before its launch.

“We have had great feedback from people who are blind and have low vision,” says Tamara Rogers, global chief marketing officer at Haleon. “They really value the increased independence that comes from being empowered to access our product information themselves.”

The new functionality was developed not just for people who are blind or have low vision, but also for those who have literacy challenges, Rogers says. “There’s a classic health industry saying which is, ‘Always read the label, always read the instructions,’ and for some, that just isn’t possible,” she says.

“This initiative will make Haleon products more accessible to people who are blind or have low vision. It will also provide more independence to people with low literacy levels. This is a great way of being able to communicate in a different way, audibly, rather than asking people to read.”

Launched in 2017, Microsoft’s Seeing AI application is an AI camera-based app that can read text such as documents or signs, describe scenes, recognize currency and even identify friends. Available in 19 languages, the app has become a multipurpose tool that is helping people who are blind or have low vision navigate their daily lives. The newly enhanced functionality helps consumers in the U.S. and U.K. read packaging details in English. Previously, there wasn’t an easy way for Seeing AI to read detailed information on packaging without a user moving a camera around the text.

The collaboration with Haleon “is really important and exciting, because Seeing AI has always been able to recognize products, but the challenge has been having access to data that is really rich, and that comes from the manufacturer,” says Saqib Shaikh, an engineering manager at Microsoft who led the team that developed Seeing AI.

“Now you can just scan the barcode and hear exactly what was written on the package. It’s directly what the manufacturer wanted you to hear. The information is really accurate, and you’re getting it all in one place. You’re not having to scan all the different sides of the package to find the bits you want. It’s all right there once you’ve scanned the barcode.

“Now you can give more information, and trustworthy information,” Shaikh says. “Haleon is leading the way in doing this.”

Shaikh hopes more companies will follow Haleon’s lead and use technology to provide audio information for people who are blind or who have low vision, or those with literacy challenges. Doing so could provide important information not just for health products, he says, but for food items and countless other things people use in their daily lives.

“I’d love to make every product out there accessible so you can just scan it, whether by bar code or some other future technology, and know what it is,” he says. “This data is out there. It’s just that all the data’s in siloes. What we were able to do here was, by the companies having the will, to break down these siloes.”

More than 3 million people in the United States and more than 2 million in the U.K. are living with sight loss, and millions more struggle with poor literacy. A study commissioned by Haleon found that 93 percent of respondents didn’t think health products were accessible enough, and almost one in five said they had taken an incorrect dosage of a product because they couldn’t read the packaging.

Haleon saw technology as a tool to address those issues, Rogers says, but wanted to build on an existing platform rather than creating new technology that people who are blind or have low vision would need to adopt. Haleon approached Microsoft about using the Seeing AI app to create audio guides for its products, and the two companies worked closely together to build out the new functionality for Haleon’s bar codes. Microsoft developed an end-to-end pipeline that allows Haleon to control the data for its consumer products, which is stored in Azure, and update information or add new items as needed.

Powell says to his knowledge, Haleon — a spinoff of GlaxoSmithKline created in July 2022 to focus on consumer health products — is the first consumer health company to provide accessible information through Seeing AI for its products.

“I think this will be really useful,” he says. “This allows us as blind people to have the same level of knowledge and understanding as someone who can read the packaging. So this is a really good step.”

Click here to load media

Amar Latif, a Scottish television personality and travel company founder, lost his sight at age 18 due to retinitis pigmentosa, a rare eye disease. Like Powell, he’s had to rely on other people to read him information about health products and medicines.

“When I lost my sight, I really struggled to know what things were — bottles, potions, medicines, toothpaste,” he says. “It’s so hard to know A, what it was, and B, how to use it. So sometimes I just had to risk it. Sometimes it just wasn’t practical to ask somebody, so you just start to take matters into your own hands and wish for the best.”

The new functionality for Haleon products using Seeing AI, Latif says, will help people like him access health product information with greater independence and privacy.

“It’s fantastic. A blind person can use their phone to scan the bar code on loads of products,” he says. “If I was to scan a toothpaste or some painkillers, not only am I able to read all of the information that a sighted person can read, but I can jump from heading to heading.”

To Powell, making information on packaging accessible to people who are blind or have low vision isn’t just a matter of protecting privacy, minimizing health risks or providing greater convenience. It’s a fundamental right.

“We need to start to think about this as a wider equality and inclusion issue,” he says. “We do have a right to that information, and now, technology can provide that. I think this is something that we should look to implement more broadly as a basic standard moving forward.”