Hello, and welcome to Microsoft Stories, a new podcast about technology and innovation.
In this episode we hear about Project Tokyo, an incredible research project from Microsoft’s Cambridge Research Lab that’s focused on exploring technological solutions for people who are blind and partially sighted.
You will hear from Principal Researcher Cecily Morrison (above; photo credit: Jonathan Banks for Microsoft), who led a group that adapted a Microsoft HoloLens headset. When a person who is blind puts on the headset, it will call out the names of the people in the room with them.
You will also hear from Hector Minto, Microsoft’s Accessibility Evangelist for Europe, about how technology is changing the way people with disabilities interact with the world, and how those people are increasingly creating their own digital solutions.
Click the play button and join us on our journey.
Click here to visit our Podcast page.
Transcript of this episode:
MUSIC INTRO
Hi, I’m Andy Trotman, Head of News at Microsoft UK. Welcome to Microsoft Stories – a new podcast looking at technology and the people who use it.
In this series, I’m trying to answer the question: what is innovation? It means different things to different people. Innovation can be as simple as adding an eraser to the end of a pencil or as complex as sending people to the Moon.
What does it mean to be innovative? How do you know you’re being innovative? Along my journey, I meet people using technology in amazing ways, and discover what innovation means to them.
Join me on my journey.
SOUND OF SOMEONE MOVING CUTLERY AND PLATES IN A KITCHEN
A boy, whom we are calling TH to protect his identity, is sitting at his kitchen table, wearing a Microsoft HoloLens headset. TH is blind, but when he turns his head and faces another person in the room, the HoloLens calls out the name of that person. TH tries this with the three people in the kitchen with him. Every time, the HoloLens calls out a different name, and every time TH laughs.
This is Project Tokyo, developed in Microsoft’s Cambridge Research Lab by a team led by this person.
CECILY: Yes, my name is Cecily Morrison. I’m a principal researcher at Microsoft Research in Cambridge, UK. And I lead the Project Tokyo team.
When it comes to innovation, there are few places in the world quite like Microsoft’s Research Labs. They are a place where researchers, scientists and experts of all kinds are pushing the boundaries of what technology can do now, and working towards creating the tech people will need in the future.
The Lab in Cambridge has helped develop innovations such as Project Silica, which involves storing data on glass, a material that’s much more durable than those we use currently.
I wondered if all the groundbreaking projects developed in that lab created more pressure to be innovative.
CECILY: I don’t think pressure is the right word. I think it creates a very generative environment. Everyone’s excited about new ideas. They’re excited about thinking about things. Everybody’s happy to sit down at lunch and just think, just have a chat, not try to reach a particular goal. And I think it’s often those chats, those unexpected conversations which can lead to something so powerful.
The start of Project Tokyo was a similar story of collaboration. People at the lab had wanted to do something with artificial intelligence using what they call “agents” – that basically means any technology that uses sensors to understand the world around it and then act on that information.
Senior staff approached Cecily, who was interested in using agents to help people who are blind. She had previously worked on Project Torino, which uses small pods that can be connected together to teach blind children how to code.
I was intrigued by the approach Cecily and her team took when attempting to innovate at such an advanced level. After all, they were trying to develop a solution that could literally change the lives of millions of visually impaired people across the world. It turns out, Project Tokyo started in a very different way to how it is now. It didn’t start out with children at all, but with Paralympic athletes.
CECILY: We then started a long process of in the beginning, we were working with Paralympic athletes and those travelling to the Paralympics, just to imagine how AI technology could augment their existing capabilities. We create AI technologies in Microsoft which are about augmenting people’s capabilities and not about replacing them. It’s much easier to imagine how we replace capabilities. But I don’t see that that’s the future of AI technology. So we had to think about how do we go about imagining this new thing. We’re going to have these extra capabilities. What do we do with them? So we started there. And then the journey took many twists and turns until we got to the kids that we are working with today.
With Project Tokyo, Cecily and her team innovated towards a solution but ended up going in a completely different direction after seeing, in her words, the power of the technology for children.
Cecily laughed at my suggestion that you can plan innovation from the beginning. She believes you can sketch out the journey of what you want to create, but the innovation happens along the way. Sometimes innovation happens because you encounter a small problem that needs to be solved, but it can also happen not because of something the creator of the technology does, but the user.
For example, Cecily noticed that when using Project Tokyo, TH would move his head away from the person he was talking with and then move it back again, so the HoloLens would repeat the person’s name. He did this every 90 seconds or so. Cecily was baffled. Surely TH knew whom he had been speaking to for the past 10 minutes? The answer was something Cecily’s team hadn’t considered.
CECILY: It could be that he was using this to refresh his working memory, because the time of your working memory is about 90 seconds to two minutes. When you can’t see, it’s something that’s very hard to refresh. If you see, it’s something you do visually, if you can’t see, you have to have a lot more processing going on to constantly keeping track of what’s around you. So TH had worked out that if he used the system, refreshed the names, it would be easier for him to keep these people around him in his working memory. And as a result, it’s easier for him to maintain his attention to these people and the topic of conversation that he’s engaged with. So this is a completely unexpected use of our system. We couldn’t possibly have imagined this when we built it.
What I learned from listening to Cecily is that when it comes to innovation, there is a chain reaction taking place. The next creator learns from those before them and adapts their technology accordingly. That happens over and over again.
Hector Minto, Microsoft’s Accessibility Evangelist for Europe, has worked in the field of assistive technology for 20 years. He has seen, first-hand, the many ways people have taken features in technology and adapted them.
HECTOR: We record meetings, and quite often I’ll go for a run in the morning and I will listen to a meeting recording rather than attend the meeting, you know if there’s a meeting that I didn’t need to go to.
SOUND OF SOMEONE JOGGING
Now, the fact that I can do that is because somebody decided to offer that screen recording option within, say, Teams. But what also happens when you record a meeting is that you automatically transcribe it for people who are deaf or hard of hearing, which is also just a very deliberate design principle that somebody put in. But then what you find out is that text that you’ve generated from that meeting can then be translated into multiple languages through translation engines. And then what you also then find out is that people can go and search through a video based on what people said, rather than what somebody called the video and what somebody tagged the video with. So in almost every instance of deliberate, inclusive design or different scenario design, you find out that that different audiences pick up on it and maybe add their bolt on to it to create their even more specific way of using that tool.
Looking back over the last two decades of his career, Hector has seen lots of innovation cross from being mainstream into the disability space. But he’s also seen innovation go the other way, too.
HECTOR: And I always say to people, jokingly, although it’s true, you know, I saw emojis before everybody else, because they’re being used in the world of communication for people who are nonverbal. Pictograms to express communication lived in the disability world first. I saw touchscreens before most people, I saw voice-controlled home automation before everybody else, because it was being applied in the world of disability first. Most new technologies come from the world of disability and that’s why it’s such an important space to be focusing on.
Xbox has a co-pilot setting that lets two people uses separate controllers but to move one character. It’s designed to help gamers with disabilities. But what the creators found was that children with autism were also using this setting, and that meant they had to communicate with the other player in a really meaningful way. That was a really positive outcome that the Xbox creators hadn’t intended when they began their innovation journey.
I’m learning that there is a fluidity to innovation in this area that’s quite unique. Most of the time that’s a good thing. There is a lot of work happening, but not all of it is helpful. Unfortunately, products are still made for disabled people that are more of a hindrance than a help.
The key is to get feedback from your target audience, especially if you’re creating a solution for a group that you’re not part of. In the case of Project Tokyo, Cecily has full sight, and she has created something for people who are blind. Gathering the opinions and input of people in the blind community was crucial in order for her team to create a solution that was helpful.
Hector says the unfortunate truth is that too many products for disabled people are still being designed without speaking to the people who will eventually use them. That’s incredible to me. I can’t think of another area that doesn’t involve its customers in the design process.
He added that Microsoft adopts an outside-in approach to innovation on disability technology.
HECTOR: Listening to people with disabilities and what they need from us drives so much innovation at Microsoft. Some of those ideas that we get come directly from feedback from our customers with disabilities, and people who use our products. So, absolutely, listening, having things like insider programmes, support desks, user voice forums where people can vote up the features they want from us in the accessibility space. I’m constantly telling people to use the disability answer desk, use the user voice forum because our engineers, they look at that data, and it’s absolute gold as to what things that they might go to next.
Take that one step further and the potential for innovation gets really exciting. What if big companies just created the tools for people with disabilities to create their own solutions and then just stepped back. Like Project Torino, which I mentioned earlier, no-code or low-code solutions are becoming popular in the accessibility space. That means people with disabilities creating apps or products for themselves and others with disabilities.
It’s the democratisation of innovation.
Here’s Hector again.
HECTOR: So I would actually go so far as to say, we’re moving to a world where we’re not just listening to what innovation we need to do, but we’re creating the toolkits and the platforms for people to go and problem solve for themselves. It won’t just be computer scientists, IT departments that will be creating these tools, it will be citizens. When citizens and diverse sets of citizens start creating solutions to their daily experiences, that’s when the innovation is going to start. And that’s going to get really, super exciting.
When you get to that point, I believe it goes a long way towards society removing the labels from assistive technology and its users. Those innovations are no longer called a solution for people with disabilities but simply a solution. Because innovation will be occurring at the heart of where the problem is, by the people who know best about what they need. There will be no third-party creator to put a label on it.
Even I am one of those third-party creators. I’m making a podcast about accessibility but I have no accessibility needs. Recording this podcast opened my eyes to so many aspects of life I hadn’t thought of before, because I’m lucky enough that I don’t have to think about them. Something simple for me, like picking out what T-shirt I was going to wear today, is a big problem for people who are blind. They often innovate by sewing different tags and labels into T-shirts so they know which one they are picking up.
HECTOR: That’s the sort of level of problem solving you have to do if you cannot see which T-shirt is in a drawer, which shirt, which tie you’re going to wear. Just through Microsoft exposing ourselves more and more, very deliberately to audiences who are problem solvers and supporting this agenda, where we really believe that people with disabilities can start writing their own software.
We’re really ambitious that people with disabilities will write their own software in the future, they will build it, they will create their own tools.
According to Cecily, people who are blind or have low vision already embrace a lot of agent technology. The Edge web browser has a feature that reads web pages out loud, for example.
SOUND OF COMPUTER READING OUT WEBPAGE
There are voice-activated speakers in homes across the world. Cecily believes that including that community in the innovation and design process will improve how humans and AI innovate together in the future.
That’s important because AI and humans working together are more innovative and productive than either working separately.
CECILY: We’d already created this solution by thinking about what the future is going to be like, and therefore we can help other teams think about – once they can see what that future is like – think about how they create new technologies to support that. So overall, I think inclusive design has been a very productive way for us to innovate. And I think it’s part of a larger form of innovation. And when we think about human-centred design, we start with people and society. And we think about how can we open up new possibilities for people. And that drives the innovation process.
Inclusivity is key for Cecily as she tries to innovate. It’s an approach that applies to the teams building the technology as much as it does to the people who will end up using it. The Project Tokyo team includes experts in computer vision, human computer interaction, design ethics, machine learning, people who run user studies and those from disability studies. There are also universities and PHD students involved, a designer and an engineer.
CECILY: I think it’s critical to be around other innovative people, but also people with different skills and different life experiences. A lot of our innovation comes from being able to draw upon a wealth of experience about how the world’s working and the more diverse our teams are, the more unexpected experience or unexpected insights we can draw from the experience of our team.
So diverse teams at Microsoft are important in order to be innovative when creating technology for people with disabilities. From a purely business point of view, ensuring your audience wants what you’re creating makes sense, too.
According to the charity Scope, there are 13.9 million disabled people in the UK. That’s a huge proportion of the population who may need custom-made solutions.
Many technology companies are rightly taking their work on accessibility seriously.
HECTOR: Accessibility has always been a really innovative space or the world of assistive technology. But I’ve got to be honest, there’s never been a time, like right now, you know, there’s a new toolkit. If you think about cognitive services in the cloud, if we think about cloud connectivity, more generally, people are looking at new tools that really empower people with disabilities on a daily basis. But also because we have this culture of inclusion, and there’s this new focus on disability inclusion at Microsoft, we’re finding that each team is picking up its own responsibilities to do something innovative in this space. So yeah, it’s a thriving subject matter right now within the world of tech.
It’s thriving so much at Microsoft that Project Tokyo has sparked the creation of another attempt to be innovative – The ORBIT Project. Its aim is to train AI systems that can recognise personal objects. If a blind person is having a cup of tea with three friends and puts their mug on the table, how do they know which one to pick up again?
SOUND OF MUGS CLINKING TOGETHER
That’s a relatively minor example, but most products are created for a mass-market, so even important things in our lives can look the same as those owned by someone else. Imagine a blind person working in the Government, and that person places their laptop containing sensitive files next to a laptop that looks the same. How can that person pick up the right laptop?
In short, the ORBIT Project aims to use AI agents at a personal level, which would have a huge impact on people’s lives. Here’s Cecily again:
CECILY: And that came from the creating process, but that creating process needing to take a small turn, and then lots of lunchtime conversations about how we could find a way to bring machine learning and AI with a really productive user experience to really push the forefront of AI technology possibilities.
It’s normal for the Cambridge Research Lab to push the boundaries of what technology can do. Every time I visit that office I’m truly amazed by the innovation that takes place there. And that’s just the stuff I know about!
When you’re operating at that level of innovation, can you ever turn it off? When you leave the office and go home for the evening, do you stop thinking about solutions that can change the world?
Well, for Cecily, it’s good to power down her brain every now and again.
CECILY: That also makes me laugh because I used to have a hard time turning my brain off. But I have two small children so they do a good job to help my brain turn off or at least turn to something else.
I think it really depends on what problem you’re solving and where you are. In Tokyo, there’s a big problem, but then there are lots of smaller problems along the way. So I’m a big fan of sleeping on things. We have the Cambridge Botanical Gardens just up the road from us. It’s one of my favourite places in the summer to sit in the sun, and just let the mind wander and that can be really helpful for me as well.
Designing technology for people with disabilities is certainly an area packed with innovation – from the companies and organisations creating products to users building their own solutions. Thanks to Cecily and Hector for chatting to me. And thank you for listening. I’m afraid that’s all we have time for today. Look out for the next episode of Microsoft Stories soon.
MUSIC OUTRO
