A most subjective prologue

I stood in a cul-de-sac in Reading, a suburb 30 minutes west of London, preparing for one of the most unique journeys of my life.

“Journey” may seem a strong word for walking a few blocks and getting on a bus. At my normal pace, it should only be 444 steps from the Tudor-and-brick-walled quietude of Tamerisk Avenue to the bus stop around the corner. But considering the deeply meaningful work happening in this small corridor of England, and the way I’ll feel after my trip (as topsy-turvy as if I’d spent the day at Six Flags), there’s nothing else to call it but a journey.

It was midday on a Tuesday. The weather was capricious, scattering raindrops across our jackets one minute and warming our faces with sunshine the next.

“OK, I think we’re ready,” said Mike Parker, a kind, bearded Microsoft user experience designer. He handed me a shiny, black smart phone. “Your phone is all ready to go, so you can just put it in your pocket. Chris, do you have her cane?”

Chris Yates, an amiable mobility instructor for the charity Guide Dogs, handed me a long, white folding cane with a rubber stopper at the bottom and quickly showed me how to sweep it from side to side, tapping the pavement in front of me as if dipping a toe into bathwater of unknown temperature. As I tried the cane, Parker placed a pair of bone-conducting headphones around the back of my skull and handed me a heavy-duty black blindfold.

Photograph of AfterShokz bone-conducting headphones, part of Microsoft's 3D soundscape technology

I was about to try a prototype of Microsoft’s 3D soundscape technology — an audio-rich experience in which the headset, smartphone and indoor and outdoor beacons all work together to enhance the mobility, confidence and independence of people with vision loss.

This project is the result of a unique partnership between Microsoft, the charity Guide Dogs, and a number of other partners including Network Rail, Reading Buses, the urban planning agency Future Cities Catapult, the Reading Borough Council and the grocer Tesco (not to mention the understanding neighbors on Tamarisk Avenue).

Once the heavy blindfold blocked all the light, my other senses clumsily shifted and my hearing went into overdrive as the headset started sending 3D audio cues directly into my inner ear.

“Uh, I hear something like that galloping coconut noise from Monty Python,” I said. The guys chuckled.

Jennifer Warnick, Jarmail Chudge and Chris Yates during a test of Microsoft’s 3D soundscape technology

The team placed Bluetooth beacons on fixed neighborhood objects to help create an information-boosted route through a London suburb.

The galloping coconuts sound seemed to be coming from a meter or two in front of me, and would become a comforting indicator of my forward progress on the correct (beacon-embedded) path through the neighborhood. (Later, lost and hungry in London, I found myself wishing the galloping coconuts could lead me to the nearest well-reviewed pub for a pie and a pint.)

As I took my first tentative steps, I noticed a second sound — a sort of sonar ping. Within a few strides the ping seemed to move to my left side (which it turns out was to let me know I was veering left toward the curb). As I corrected, the pinging sound moved back to center as the clip-clops continued to nudge me forward. Periodically, a voice offered turn-by-turn directions, nearby points of interest (“Chiropractor, about 10 meters”), transportation updates (“No. 9 bus is approaching”) and even polite warnings (“Be aware: This is a main road”). Because the headphones didn’t cover my actual ears, I could also listen for environmental noises. I’d never before had an audio experience like this — its richness helped me visualize the neighborhood around me while its immersiveness gave me more confidence with every step.

Reading, England Restaurant Vector Illustration of a Reading, England restaurant is programatically drawn on the screen to represent the expanding environment of of Microsoft’s 3D soundscape technology
Reading, England Storefront Vector Illustration in Gray and Blue-green Reading, England storefront is programatically drawn on the screen to represent the expanding environment of of Microsoft’s 3D soundscape technology
Reading, England Bus Stop Vector Illustration of a Reading, England bus stop is programatically drawn on the screen to represent the expanding environment of of Microsoft’s 3D soundscape technology

Eventually, about halfway through the walk, I relaxed enough to carry on a conversation. By the time I reached the bus stop, I was chatting away with the team from Microsoft and Guide Dogs, even as the headset beamed route updates and points of interest to my inner ear.

I was so excited to get the hang of it that I was reluctant to remove the blindfold and headset once we reached Reading Station. Parker and Yates said this is a common reaction from people who have made the journey, visually impaired and sighted alike.

Commuter Bus Driving in the Rain Vector illustration of a Reading, England commuter bus drives through capricious, scattered raindrops, carrying the Microsoft Team

“There hasn’t been this much magic in
the British suburbs since Harry Potter
was dropped at 4 Privet Drive.”

“People want to take it and start using it now,” Parker said. “We have to explain it’s just a prototype, that this is just the beginning.”

I’m not sure if I could make my way across my own living room blindfolded, at least not without some bruising, and yet I’d just traveled across an unfamiliar city relying primarily on a cane and a few well-placed, 3D sounds. Where I anticipated feeling vulnerable and anxious in the blindfold, I ended up feeling strangely super-powered wearing the headset, like some sort of dry-land dolphin.

“Man,” I thought to myself as the train to London pulled away from Reading Station, “there hasn’t been this much magic in the British suburbs since Harry Potter was dropped at 4 Privet Drive.”

The prototype headset is, rather charmingly, held together by red electrical tape. The Windows Phone is off-the-shelf. The Bluetooth beacons sending information to the headset and phone look like plastic, Smurf-colored Oreo cookies, and they are zip-tied to poles and lamp posts. Yet somehow, the delightfully DIY experience is vastly more than the sum of its parts. It may not look particularly magical, but we all know how deceiving looks can be.

City Skyline with Billboard Vector illustration of a city skyline with a billboard to frame the independence day video. Read the script (Opens in a new tab)

Chapter 2:
‘Here we go!’

There is a particularly memorable scene in the French movie “Amélie” in which the title character, a shy waitress with a flair for helping others, sees an older man with a white cane waiting to cross a busy street. “Let me help you. Step down,” she says, taking his elbow. “Here we go!”

Lollypops in a Bakery Window Vector Illustration of Lollypops in a bakery window on a busy street in Paris from the 2001 movie, Amelie.

And they’re off, moving quickly through the neighborhood as Amélie provides a running commentary: The golden bust of a horse hanging over a shop is missing an ear; that laughing is the florist, who has crinkly eyes; there are lollipops in the bakery window.

Ham Butcher's Diagram Vector illustration of a ham butcher's diagram in a butcher window on a busy street in Paris from the 2001 movie, Amelie.

“Smell that! They’re giving out melon slices. We’re passing the park butcher, ham is 70 francs. Now the cheese shop — picadors are 12.90. Now we’re at the kiosk by the metro. I’ll leave you here. Bye!”

Ticket from a Paris Metro Kiosk Vector illustration of a ticket from a Paris metro kiosk from the 2001 movie, Amelie.

After she leaves him, the man turns to the sky, a look of sheer joy on his face. While he has no doubt traversed that street many times before, Amélie’s observations made it a different kind of trip. It was vivid.

Jennifer Bottom sitting in the window seat using her phone to access Microsoft 3D soundscape technology on a train from Reading to London

Jennifer Bottom, 26, uses Microsoft 3D soundscape technology on the train ride from Reading to London.

“There are quite a few technology offerings out there which certainly claim they’re going to change the lives of blind people — that it will be the same as if you could see, which of course isn’t true,” said Jennifer Bottom, a 26-year-old who works in tech support for a software company.

Bottom and her guide dog, Archie, and other Guide Dogs users were a critical part of the technology’s pilot program.

She said she would definitely use the technology when it comes to fruition — not as a replacement for Archie but as an additional mobility aid.

“I’m quite a cynical person. It took me a few minutes to get used to the technology and to trust the feedback,” Bottom said. “Overall, I was very comfortable with it. You can definitely tell that visually impaired people have been consulted, not just a bunch of executives.”

Terry Brewell and his guide dog, Zed, at a London metro turnstyle, while testing Microsoft 3D soundscape technology

Terry Brewell, 69, said the new technology is a comforting mobility aid much like his guide dog Zed.

While Bottom has been visually impaired since birth, Terry Brewell, a 69-year-old retiree of the phone company, lost his sight more gradually. He had a fair number of mishaps — misjudging curbs and once even tumbling down some stairs at a busy train station — before he finally got his vision re-assessed and was paired with his first guide dog. That was nearly 40 years ago.

“I was sad to realize it was getting significantly worse,” Brewell said. “But when I got my first guide dog, it took a lot of the stress away.”

Brewell said using the Microsoft 3D soundscape technology was comforting in a way similar to traveling with Zed, the black Labrador retriever sitting near his legs on the train as we talk.

“It makes you feel more confident. You get advance warning of what’s ahead, or what your surroundings are,” Brewell said. “One of the hardest parts is being confronted by unexpected situations. Like where I live they’re rejuvenating the town center, and it’s quite a mess.”

Microsoft designers worked incredibly closely with Guide Dogs — its employees, mobility experts and users like Bottom and Brewell — to genuinely understand the challenges of traveling to and fro with vision loss. The engineers and designers from Microsoft and mobility experts and users from Guide Dogs spent countless hours in the field together. In rain and wind, they patiently tried various half-baked ideas, experimented with different approaches to hardware and software, and gave essential feedback to help shape the technology every step of the way.

“It really was a deep dive into the whole customer experience, which has continued throughout this three-year process and still continues today. We haven’t stopped,” said Jenny Cook, head of strategy and research for Guide Dogs. “The possibility of what this could be was exciting and still is. We’ve only done a concept, it’s still pure research and development, but the possibilities are endless and the impact is incredible if we get this right.”

It takes weeks for a mobility trainer to help a user learn a short route, say from home to a bus stop, and even familiar paths can yield unanticipated difficulties — silent cyclists, disruptive construction, noiseless smart cars, hanging tree branches, tipped-over garbage bins, delayed buses. For many, travel becomes a stressful and overwhelming prospect.

“In the U.K., there are about 180,000 people who rarely if ever get out and about. That’s a massive social issue in its own right,” said Richard Leaman, CEO of Guide Dogs. “That was really the heart of our new strategy. Supporting blind people with 5,000 guide dogs is really not scratching the surface. Introducing new services, like the use of technology, is a vital ingredient to change. If our major cities in the U.K. had sensor-enriched zones where this capability worked, I think it would be an incredible achievement. Beyond that, I expect this to go global, as I’m sure major cities around the world will want the same. It’s like an awakening.”

“In the U.K., there are about 180,000 people who rarely if ever get out and about.”

Chapter 3:
The mother
of invention

Whether or not he knew it when he started talking about using technology for those with vision loss, Amos Miller is the perfect person to challenge the status quo.

Miller, who grew up in a kibbutz in Israel, was diagnosed with retinitis pigmentosa (RP) at age 5. There was no telling when the rare, genetic disease would start to take his eyesight; RP has a reputation for unpredictability. His vision could have started to deteriorate immediately, or he could still see well enough to drive at 60 — there was no telling.

“I started university fully sighted, but finished not being able to read anything. Not the white board, not the books, not the computer screen,” Miller said. “I actually did go through some therapy at university to start calling myself a blind person. That wasn’t the purpose of it, but it was one of the results. I learned that life could be much better and easier once you accept it, and it did become much better.”

Several years later, after graduating with a degree in computer science from the Israel Institute of Technology, Miller and his new wife (a South Korean who’d been living in Israel) moved to London (where he had dual citizenship, as his mother is British) to broaden his horizons. There, he worked IT jobs and they “lived the Yuppie life.” He got his first guide dog. He earned an MBA from London Business School and, in 2007, he started work for Microsoft. A year later, he saw an advertisement in the Guide Dogs newsletter seeking candidates for the charity’s board of trustees. The oldest and largest of its kind in the U.K., Guide Dogs helps thousands of people with visual impairments by providing them with specially trained dogs and mobility training.

Guide Dogs wanted someone with management experience and, preferably, a guide dog. Miller wanted to take his new MBA out for a spin for some leadership experience.

“I didn’t go in with a mission to change Guide Dogs. I went in with a mission to find out what a trustee was,” Miller said, laughing. “As someone blind with a passion for technology, I helped with some initial thinking around how technology fit into the life of a blind person, and what role Guide Dogs could play in that.”

Miller worked closely with Leaman, Cook and others from Guide Dogs. All of them could see it was a time of immense opportunity and transformation for the 80-year-old charity, originally founded between the first and second World Wars to give seeing-eye dogs to returning soldiers who’d lost their eyesight in battle. A majority of the organization’s 5,000 users don’t use anything more high-tech than a guide dog or a white cane for mobility, Cook said.

  • Jenny Cook, head of strategy and research for Guide Dogs. Photographic portrait taken on a glass staircase at Paddington Station
    “We’ve only done a concept, it’s still pure research and development, but the possibilities are endless and the impact is incredible if we get this right.” Jenny Cook,
    Head of strategy and research for Guide Dogs
  • Dan Hill, Executive director of Future Cities Catapult. Photographic portrait taken seated on a staircase at Paddington Station
    “We know a lot about how to design and build in a human-centered way, but it doesn’t always happen. Just think about what we could do if we incorporate the capabilities of technology — not for technology’s sake, but for human sake.” Dan Hill,
    Executive director of Future Cities Catapult
  • Richard Leaman, CEO of Guide Dogs. Photographic portrait taken on a busy London street
    “We’re an organization that four years ago was just about dogs, and we’re here talking about incredible, landscape-changing stuff. We’re making history.” Richard Leaman,
    CEO of Guide Dogs
  • Chris Yates, Customer experience lead for Guide Dogs. Photographic portrait taken at the entrance to the London Underground
    “Cities need to be built for absolutely everyone, but no one’s going to knock them down and make them accessible. What we’re doing is using existing infrastructure and working around it with technology.” Chris Yates,
    Customer experience lead for Guide Dogs

So could life for a person with visual impairments be different or better with technology in its fabric? If so, how?

As Miller, Leaman, Cook and others from Guide Dogs contemplated these big questions, Miller asked his Microsoft colleague Jarnail Chudge to help host a brainstorming workshop for the charity. Miller and Chudge had worked together on a couple of projects, and Miller appreciated Chudge’s ability to “tease out of people in a room their hopes and aspirations" rather than diving into tech talk right away. Little did Miller know that it was the beginning of an unlikely friendship and a powerful duo.

“I respected him very much as a user experience architect and a very humble and creative guy. We’re so different, but now I think we’ll be friends forever,” Miller said. “We both don’t see obstacles. We’re both the kind to push forward regardless of the situation. We’re both blind to challenges and risks.”

But where Miller is a big-picture thinker, Chudge is relentless in his execution.

“Put it this way: If this had been left to me, I would still be thinking about it,” Miller said.

The son of hard-working Indian parents who immigrated to England, Chudge manages to be at once fiercely passionate and genially mild-mannered. One workshop led to many more. For Microsoft and Guide Dogs, the process of building a partnership was anything but speed dating. In time, “this theme around mobility and independence started to emerge, and our collective energy started to coalesce a little bit,” Chudge said. Microsoft and Guide Dogs decided to produce a concept video as a vision statement for the partnership, something to show real-life scenarios and explore the possibilities of using mobile devices and 3D sound to help enhance the environment and experiences of a person with visual impairments. In the 15-minute video “A Family Day Out,” a man with visual impairments decides to take his son and father-in-law on a day trip into London, including to a museum, where they see a life-size model of a blue whale. The whole thing is an echo of many Guide Dogs users, and even Miller’s own desire for more confidence and freedom when he’s out and about with his daughter.

“How do you convey something so
moving as a whale in words?”

“How do you convey something so moving as a whale in words? How do you describe it so that the blind person visiting gets a shiver down his spine as well as those who can see it?” Miller said.

Meanwhile, the video made its way to Microsoft headquarters in Redmond, Washington, which is how Jenny Lay-Flurrie, the company’s senior director for accessibility, first learned of the project. She would become the first of a handful of high-profile, vocal champions of the project at Microsoft.

“It’s become an egoless, boundary-less, beautiful team. There are champions, and fuse-lighters, and fixers, and people who will knock anything down that gets in the way. I can honestly say this has been the most enjoyable, impactful, energy-giving project I’ve been on at Microsoft and potentially in my career, and I’ve gotten to do some pretty cool things,” Lay-Flurrie said. “But Amos Miller is the inspiration. He’s the motivation — the rock that keeps us grounded to the purpose.”

To Lay-Flurrie, the purpose is most shockingly illustrated by one statistic. Did you know, she said, that in the United States about 65 percent of people with vision loss are unemployed?

“There’s no better meaning than helping someone to be independent in their life,” she said. “Everybody has that right, but not everybody has that capability. But I think with technology we can make it happen for everyone.”

Chapter 4:
It takes a village

Miller and Chudge were determined to find a way to make the concept technology in their video come to life. But to create a pilot program would take a groundswell of support and some considerable funding. Around that time the concept video won a top prize at Microsoft’s annual Ability Summit. Satya Nadella, Microsoft’s CEO, who was then executive vice president of the company’s Cloud and Enterprise group, presented the two with the award.

“When Satya walked off stage, Jarnail zoomed behind him. Literally,” Miller said.

“It’s true. I was tracking his movements, and as soon as I saw him leave I made my excuses and ran to catch him,” Chudge said. “I thanked him for the award, and asked if he’d be willing to give us an hour of his time to talk in more detail about what we were trying to do. He said, ‘Yep, that’s fine, just drop me a note.’”

It took Chudge two weeks to compose the email. “I lost count of the number of drafts,” he said. “But once I finally sent it, he responded immediately.”

Nadella listened intently and afterward, asked a few people to take a deeper look at the project. Included in that group was Dave Campbell, chief technology officer for Microsoft’s Cloud and Enterprise group. Campbell then roped in a couple of people from Microsoft Research — Rico Malvar, chief scientist and distinguished engineer, and Bill Buxton, principal researcher. The team grew quickly, especially considering it was a passion project, something people were making time for above and beyond their regular workload.

“The purpose and meaning of this project meant it was so incredibly easy to enlist people — it just sort of tugged at them,” Campbell said.

Campbell was particularly excited by the project because of the opportunity it presented to practice “design empathy.”

“I’ve been on this design empathy kick for close to 10 years, and I think it’s really important to the company right now,” Campbell said. “We were born as a company of engineers building products for other engineers and geeks. That’s no longer the case. This notion of design empathy, or being able to build something for someone who is not like you — it’s much, much harder. You really need to adopt a perspective other than your own to pull it off.”

Jarnail Chudge stands on Tamerisk Avenue with a pair of bone-conducting headphones during field testing for Microsoft’s 3D soundscape technology.

Jarnail Chudge stands on Tamerisk Avenue where the field testing for Microsoft’s 3D soundscape technology has taken place.

What started off as a small passion project far removed from Microsoft headquarters has shown Campbell and others the importance in avoiding subject-area silos and “unlocking the immense, aggregate power of Microsoft.” Campbell was so inspired he wrote an internal memo on the project, calling it out as a best practice.

“We have been guided by two high-level principles,” Campbell wrote. “One, focus relentlessly and empathically on the customer problem to be solved. Two, unlock the vast resources of Microsoft in pursuit of number one.”

The project’s timing was stellar as well, landing in Redmond in the midst of a hacking culture renaissance, which meant instead of fretting about what domain the project belonged in, the group was able to “reach into the pantry of Microsoft” to access any research or technology available that could help.

“That rarely happens, but is an experience we need to develop going forward,” Campbell said. “There are projects that will cut across hardware, Windows Phone, Windows OS, Bing and Microsoft Research. The hacker culture says if we don’t have the answer, we’re going to find it using whatever duct tape and bailing twine we can, and we’re going to put this together and make it work for real.”

Buxton, a leading expert in human-computer interaction, is also a musician. He was drawn to the project after hearing about its complex audio challenges, including trying to build a rich 3D soundscape. However, Buxton’s favorite part of the project isn’t the deep technology it employs, but that for users there will be a moment when the technology disappears altogether.

“The hacker culture says if we don’t have the answer, we’re going to find it using whatever duct tape and bailing twine we can.”

“The best technology is invisible. It just lets me get on with my life,” Buxton said. “When an interaction with technology is as it should be, the user is not there as an equipment operator, but as a human being. They are not walking down the street operating technology, but with the intent to go to work, or get fresh air, or exercise.”

Buxton compared using the 3D soundscape technology to wearing a pair of sneakers.

“I never think about my shoes as mobility devices, but it’s the same kind of thing. They just enable me to do what I want without getting in the way,” he said.

Along with troubleshooting the experience’s 3D technology, the team knew that users would likely have one hand occupied by either a cane or a guide. This meant it was also critical for users to have a way to hold the smart phone but use it with the same hand. The team’s solution? One-handed navigation. By holding a thumb down on the screen and moving it up and down, a user can navigate between options. By releasing the thumb, the user can select something. While this technology was designed specifically for people with vision loss, it could benefit everyone, whether it’s a tourist in a foreign city or someone driving a car through their own hometown who needs to keep their eyes on the road and hands on the wheel, Buxton said. Anyone who has walked around with a cup of coffee or a bag of groceries could appreciate the phone’s one-handed navigation. These are not only examples of design empathy, but universal design — something like an audiobook or a ramp into a building or a handle doorknob originally designed for the benefit of a few but actually of benefit to many.

For Buxton, the key to great design is simple — understand and design for the needs of highly specialized users, and you’ll often end up making something good for everyone.

“How many times have we used one of those ramps instead of the stairs when we’re dragging luggage behind us, or used an elbow to open a door handle when our hands are full,” Buxton said. “For me as a designer of interaction whose focus is always about the quality of the human experience, I found out very early on that if you want to understand something, you go to the extreme cases and try to understand things at the edges. In nearly all cases, what you learn people need while you’re there will also apply to the general population.”

The end result of the collaboration is thoughtful design and a genuinely intuitive experience, Leaman said. When he became CEO of Guide Dogs, he found that seeing a guide dog owner with their dog for the first time made the hairs on the back of his neck stand up. The same thing happened when he recently made the journey through Reading blindfolded with a white cane and the technology his team and Microsoft’s spent three years to develop.

“There were many hurdles to overcome, but the Microsoft-Guide Dogs relationship was staggeringly good,” Leaman said. “What we’ve done is create a device that is what users actually want and will actually work for them. The accuracy and information flow far exceed everything that’s out there. We’re an organization that four years ago was just about dogs, and we’re here talking about incredible, landscape-changing stuff. We’re making history.”

Chapter 5:

“What color shirt am I wearing today?” Miller asked, smiling.

I studied his shirt for a moment. “Royal blue plaid with navy and white.”

“And my slacks?”

“Hmm. Kind of a brownstone.”

We were standing in perhaps the single busiest spot in Seattle, rocks in a surging river of tourists moving into Pike Place Market. The corners of his mouth turn up slightly as he takes it all in — the holler of fishmongers tossing halibut, the slap of the fish being caught, the young busker and his slightly out-of-tune fiddle struggling through some Vivaldi, the smell of made-to-order donuts and fresh-cut dahlias.

“It’s lovely,” he said. “There’s a lot going on. I like it.”

Icon Neon Sign at Pike Place Market Vector illustration of the icon neon sign at Pike Place Market in Seattle.

It was his first time visiting the market, one of Seattle’s biggest tourist attractions. It had been a week since I experienced the sound-rich journey he helped create on the other side of the ocean, and now it was my turn to play tour guide.

It’s not hard to imagine Pike Place Market chock full of sensors and you wearing a headset on a future visit here, I told him. Miller agreed, and in fact he dreams of not only an expanded breadth for the technology but an increased depth that will offer more vivid descriptions of the surroundings.

“This is Version 0.1,” Miller said, gesturing to the prototype headset. “Someday I want the language we use to be rich and creative — to use metaphors and prose to help me build a picture of what’s around me.”

Amos Miller, pioneer of Microsoft’s 3D soundscape technology, experiences Pike Place Market in Seattle for the first time.

Amos Miller said this experience gives him hope that in the future, technology will be more thoughtfully designed.

At a coffee shop nearby, Miller explains some of what he’s learned so far. One big realization? When you design something, if you care deeply about people with special circumstances using it you will inevitably be caring for everyone who uses the experience.

“That actually gives me hope that technological products down the line are going to be fun for me to use, and not a drag, because I love technology.”

Whether it’s a building design or a phone, accessibility, in a way, is always an afterthought, he said.

“The experience is sub-optimal for me as a blind person using anything, but even when I can get access to something the feeling I get is that no one cares about my experience,” Miller said. “Can I use it? Yes. But is it going to be fun for me? If it’s good for the one percent, it’s going to be good for the 99 percent.”

Just like someday there will be a character with a disability in a movie where the plot doesn’t involve their disability.

“That’s got to change. Can there be a mother in a sitcom who is blind, but it not be all about her being blind?” Miller said. He sighed.

“I just happen to be blind. And I happen to have a soft spot for the subject,” he said. “Without question, my inspiration comes from my daily frustrations.

“Or should I say opportunities.”

Headset photo by Mike Evans / © Microsoft All other photos by Brian Smale / © Microsoft