Natural User Interfaces: Voice, Touch and Beyond

Editor’s note – Dec. 22, 2010 –
“Project Natal” referred to in this article is now Kinect for Xbox 360.

REDMOND, Wash. – Jan. 6, 2010 – Bill Buxton first used a computer in 1971. It changed his entire perspective on life.

Even four decades ago, Buxton could picture a future enhanced by technology. Eventually he came to dream about humans and computers having close interaction – being able to operate a computer by gesturing at it or by touching it, or having a computer recognize your voice and face.

“I’m excited more now than I have been since I’ve been in the business because I can taste it now,” says Buxton, a principal researcher at Microsoft since 2005. “Stuff I’ve been working towards and thinking about and dreaming about for 20 or 30 years is now at the threshold of general usage.”

Touch, face- and voice-recognition, movement sensors – all are part of an emerging field of computing often called natural user interface, or NUI. Interacting with technology in these humanistic ways is no longer limited to high-tech secret agents and Star Trek. Buxton says everyone can enjoy using technology in ways that are more adaptive to the person, location, task, social context and mood.

He sees a bright future in which entire “ecosystems” of devices and appliances co-exist with humans in a way that makes life better for people. Microsoft, with researchers like Buxton, is a leader in developing these new, more natural ways of interacting with computers. The company will showcase some of this technology at the 2010 International Consumer Electronics Show (CES) in Las Vegas this week.

Project Natal, which turns game players into human controllers, is among the most high profile examples of the coming shift in technology, Buxton said. Microsoft announced at CES this week that the Xbox gaming device will be available in stores this coming holiday season.

Project Natal is the code name for an Xbox 360 add-on that incorporates face, voice, gesture, and object recognition technology to give users a variety of ways to interact with the console, all without needing a controller. It’s a “delightful” new way to spend time with friends and family playing games, watching TV shows and movies, and listening to music, says Robbie Bach, president of Microsoft’s Entertainment and Devices Division.

Bach says Project Natal and other NUI-related products will offer more natural ways to interact with video games, computers and other technology.

“For me, when people talk about touch and voice technologies, or anything related to Natural User Interface, it all comes back to what’s most natural for the users,” Bach says. “That’s why you’ll see a variety of user interfaces that are considered natural, because each one is tuned to the environment in which it operates.”

The holiday 2010 release of Project Natal will come exactly one decade after the first Xbox console hit the shelves in the holiday season of 2000.

“Natal is a next-generation experience that we’re actually delivering this generation,” says Aaron Greenberg, director of product management for Xbox 360. “And they don’t even need to buy a new console.”



January 05, 2010
August de los Reyes, principal director of user experience for Microsoft Surface, says says natural interfaces are just the latest in a long line of evolving human-computer interaction.

Project Natal and other Microsoft-focused NUI projects represent a fundamental shift in the way people can interact with technology. The term Natural User Interfaces actually describes a wide-ranging category of technology that is perhaps most easily identified by what it lacks – the traditional methods of input including mice, keyboards, and controllers.

The goal of natural interfaces is not to make the keyboard and mouse obsolete, says August de los Reyes, principal director of user experience for Microsoft Surface. Instead, NUI is meant to remove mental and physical barriers to technology, to make computing feel more intuitive, and to expand the palette of ways users can experience technology.

Whether it’s a receptionist and patient at a doctor’s office separated by a large computer monitor, or a family in a living room sitting together in silence, parents immersed in laptops and kids texting away on cell phones, technology is increasingly creating situations de los Reyes calls “connected but alone.”

“Technology today isolates people,” de los Reyes says. NUI and Microsoft Surface are “almost anti-technology solutions.”

A member of the Advanced Studies Program at Harvard University Graduate School of Design, and a former visiting associate at Oxford, de los Reyes has become a leader in the field of finding new and intuitive ways to interact with computers.

He says natural interfaces are just the latest in a long line of evolving human-computer interaction. In the last few decades as computers became widely used, early on there was the command line interface (CLI) with its flashing cursor calling on users to type in commands. Then came the graphical user interface (GUI) and its point-and-click mouse and desktop with icons and windows. Both interfaces were revolutionary in their time, and natural interfaces are the next step, de los Reyes said.

Microsoft is “absolutely leading” in the category of natural interfaces, de los Reyes says.

Microsoft has released, and is continually developing, a number of products that incorporate touch, gestures, speech, and more to make user-computer interaction more natural – more like the way humans interact with each other.

It’s a video game that a grandmother can play with her grandson, using intuitive body movements to compete rather than having to learn to use a controller, as with Xbox’s Project Natal.



January 05, 2010
Robbie Bach, president of Microsoft’s Entertainment & Devices Division, gets into the action while demonstrating Project Natal.

Or an in-car communications and entertainment system such as Microsoft Auto’s Ford SYNC that responds to a driver’s voice commands, playing favorite songs or answering text messages so drivers can keep their eyes on the road, hands on the wheel, and mind on their driving. Kia Motors also has announced this week its new UVO in-car entertainment system, which the car company developed with Microsoft.

And it is a table that acts as a collaborative massive multi-touch-screen computer, such as Microsoft Surface, or a voice-enabled Windows phone device, or a Windows 7 laptop that lets users navigate files or the Web using their fingers or a pen tool.

Though the human-computer relationship is becoming more personalized, it’s also becoming more personally contained. The Pixar movie “Wall-E” darkly portrayed one possible version of the future in which technology usurped even the most basic human interactions, with humans moving about in high-tech chairs that meet their every need, including communication. Even if the person they are communicating with is sitting right next to them.

Though the movie’s message was a warning about technology surpassing humanism, it’s a future that’s not necessarily out of the question yet, says Buxton.

“Without informed design, [technology] is far more likely to go bad than good,” Buxton says. “It’s too important and plays too large a part of our lives to leave these things to chance. The only way it’s going to come out right is if we really work hard on understanding that … it’s about people, it’s not about technology.”

Technology for the sake of technology doesn’t interest Buxton. What interests him is when technology takes a more subordinate position – the supporting actor to humankind’s starring role.

Buxton has won a number of awards and honors for his work, which advocates innovation, design, and “the appropriate consideration of human values and culture in the conception, implementation, and use of new technology.” He also frequently teaches and speaks on the subject, and his writings include a book and regular column for BusinessWeek magazine.

“It’s not about interface design, it’s about ‘out of your face’ design,” Buxton says. “How do I get the technology out of my face so I can focus on the stuff that’s of interest of me – the material I’m reading, the film I’m viewing, the person I’m talking to, the problem I’m trying to solve and doing so in a way that brings unexpected delight.”

But creating a more natural relationship between user and technology is not merely a matter of simply removing mice, keyboards, buttons, and knobs, or adding new input methods such as speech, touch, and in-air gestures.

“The days are over where a one-size-fits-all interface is appropriate, or even acceptable,” Buxton says.

Some technology, although advanced, is not appropriate or natural in certain situations. For example, text messaging is widely used, but driving or walking while texting is difficult – even dangerous. Speech-recognition technology works well for driving or walking, but works poorly on an airplane where privacy is important, or in a noisy, crowded restaurant.

“The trick of elegant design is making sure you do the right thing the right way for the right person at the right time. What’s right here may be wrong there,” Buxton said.

Microsoft leaders say no other company is as well situated to create new user interfaces across a range of devices and contexts. “These are all pieces of a larger puzzle that we are methodically trying to solve in this emerging field,” Buxton says.

Best of all, Buxton notes, after so many years of dreaming about the true potential of computers, “I’m still around to touch and play with it.”

Related Posts