Smiles beam and walls blush: Architecture meets AI at Microsoft

Redmond, Washington and Ithaca, New York – Jenny Sabin is perched high on a scissor lift, her head poking through an opening of the porous fabric structure that she’s struggling to stretch onto the exoskeleton of her installation piece, which is suspended in the airy atrium of building 99 on Microsoft’s Redmond, Washington, campus.

Momentarily defeated, she pauses and looks up.

“It’s going to be gorgeous,” she says.

“It” is a glowing, translucent and ethereal pavilion that Sabin and her Microsoft collaborators describe as both a research tool and a glimpse into a future in which architecture and artificial intelligence merge.

“To my knowledge, this installation is the first architectural structure to be driven by artificial intelligence in real time,” said Sabin, principal designer at Jenny Sabin Studio in Ithaca, New York, who designed and built the pavilion as part of Microsoft’s Artist in Residence program.

The two-story structure, made of 3D printed nodes, fiberglass rods and fabric digitally knit with photoluminescent yarn, uses AI to translate anonymized data about facial expressions, noise, voice tones and language into a choreographed dance of color and light.

By using art and architecture to visualize information collected by microphones and cameras placed at different locations in the building, Microsoft designers and researchers hope to stimulate thinking about AI in our lives through interactive architecture.

“Artistry, creativity and humanity play an important role in technical innovation,” said Eric Horvitz, director of Microsoft’s research organization and chair of the company’s Aether Committee, which focuses on the responsible development and deployment of AI technologies, including issues around sensitive uses of AI, biases and fairness of AI systems, and human-AI interaction and collaboration.

The Artist in Residence program, he explained, was set up to invite artists to explore ideas at the intersection of art and computer science with Microsoft’s researchers and engineers and, more generally, “to stimulate joyful creation and out-of-the-box thinking across our organization.”

“Jenny’s creation,” he added, “is an embodiment of possibilities, expectations and anxieties about the rising influences of machine learning and pattern recognition technologies that are permeating the world in interesting, beautiful – and at the same time potentially invasive and concerning – ways.”

‘A living breathing thing’

The project, called Ada, weighs about 1,800 pounds.

The exoskeleton contains 895 unique and custom 3D printed nodes that connect 1,274 fiberglass rods into a web of hexagons that form the rigid, ellipsoid-shaped pavilion.

“There are a lot of parts to this,” Sabin deadpanned late one night over a plate of take-out Thai food during the project’s weekslong, painstakingly manual installation.

Photoluminescent textiles digitally knit into a porous luminous network of cells and cones are bolted onto the exoskeleton and extend inward to create a soft interior wall, giving Ada a honeycomb feel.

A one-story tall tensegrity cone wrapped in nylon mesh strung with striated fiber optic cables hangs through Ada’s core and provides compression forces to hold the entire structure in coordinated tension.

Cameras and microphones at different locations in building 99 collect anonymized data that AI algorithms translate into shifting intensities of color and light that are displayed through addressable LEDs woven into Ada’s textiles and through stage lights that surround the installation.

“It is a living, breathing thing, and it is at the heart of the building. How does that change people’s psychology about the space they dwell in and how they impact that space, and vice versa?” said Asta Roseway, a principal research designer in the urban innovation initiative at Microsoft’s research lab in Redmond who runs the Artist in Residence program.

Eric Horvitz, left, director of Microsoft’s research labs, and Jenny Sabin, right, principal designer at Jenny Sabin Studio, an Ithaca, New York-based architectural design firm, and Microsoft’s artist in residence, discuss the installation of her project Ada. Photo by John Brecher for Microsoft.

Inspiration and disruption

Roseway, who co-launched the Artist in Residence program in 2015, cozied into a couch earlier this year in the airy nook of building 99’s atrium that Ada would eventually fill and explained with an ear-to-ear grin that when people smile at Ada, Ada will metaphorically smile back at them.

The installation of Ada comes as the next wave of AI – embedded intelligence – is moving from research labs into products and services that people encounter every day. That, in turn, is raising questions and concerns about the integration of AI into our lives.

“Where does this go? What else does this lead to? How else can this evolve? Would this work for something like a hospital where people need to feel calm and better?” Roseway said.

Previous projects in the Artist in Residence program have encouraged reflections on the future of food at a time when enhanced sensing capabilities enable humans and plants to communicate, explorations of technologies that open visual arts to people who have limited vision, and clothing that responds to personal hydration levels.

“The program is really meant to inspire and disrupt ourselves, to get us out of our bubble and the way we are thinking about a certain piece of technology,” said Mira Lane, a Microsoft partner director focused on AI ethics and society, who started collaborating with Roseway after writing a white paper about the value of artists to the corporate environment.

For example, she said, designers and engineers often create a technology with a specific business use case in mind. When artists are brought into the development cycle, they’re prone to exploit the technology, break it and re-wire it in novel ways that can open eyes to potential impacts, risks and avenues for further exploration.

Lane and Roseway met Sabin at a fundraising event for the University of Washington’s art school in the spring of 2017. Sabin, who grew up in the Seattle area and graduated from the University of Washington, was invited to give a keynote talk about her work innovating new materials and forms inspired by nature that take architectural design away from hard 90-degree angles.

“There was something so inherently refreshing about that vision,” said Roseway. “Wouldn’t it be fabulous to push out a narrative where some of our future dwellings could be much more in sync with the natural shapes of the world and have intelligence embedded in them?”

An 18-month conversation ensued about what a shared project with Microsoft researchers could look like, said Sabin. Early discussions focused on creating a human-centric project that would reveal data in novel ways, surfacing the types of questions around bias and privacy that Lane’s group on ethics and society explores.

“There are many intangible structures and spatial aspects of data that we can’t really see or feel or understand without it meeting a different type of material interface,” said Sabin. “What is super exciting to me is that the project itself can facilitate, at a fundamental level, research.”

Jenny Sabin, Microsoft artist in residence, provides a tour or her lab at Cornell University in Ithaca, New York, where she is a professor of architecture. Photo by John Brecher for Microsoft.

Teaching computers to better understand people

Microsoft researcher Daniel McDuff is researching technologies that give machines the ability to accurately sense people’s emotions. That could allow an intelligent assistant to do things such as recognize when a patient skipped medications and alert a caregiver.

Computers that sense emotions could also transform gaming, accessibility and architecture, he noted.

His platform, which drives Ada, picks up on cues that are correlated with emotion. Algorithms then turn the data into numbers that represent gradients of sentiment from negative to positive and mild to intense. Ada, in turn, presents the data through color and light.

McDuff and his colleagues will store the deidentified numerical data – stripped of all video, audio and text to comply with Microsoft’s privacy requirements – collected from Ada for three years. They’ll use it to study research questions such as how weather patterns and current events impact our facial expressions, voice tones and language and to explore how patterns of behavior change throughout the day.

“I’ve looked at my own data,” noted McDuff. “On average, I smile about twice as frequently in the morning than in the evening. I didn’t know that. That seems like a lot. Hopefully this project will make us aware of some of those types of patterns.”

Participation in the project is optional and Microsoft’s privacy team was consulted to ensure appropriate disclosure and protections via anonymization. McDufff said employees can avoid engaging with the sensors by entering the building through specific doors and using the ample kitchens, common rooms and meeting spaces detached from the project.

As people opt in to the project – employees in building 99 can also install the system on their local computers and choose when to turn it on and off – we’ll begin to better understand how we impact each other, McDuff noted.

“Collectively, we decide how we want the building to be feeling that day,” he said. “We each have control over ourselves, and we have control by the way we treat and interact with others. If I am friendly to everyone around me and I encourage them to smile, I will have a bigger impact on that visualization than myself alone because I will cause each of their sensors to pick up a smile.”

Recognizing people’s expressions is similar to other work in AI aimed at better understanding people, including efforts to understand people’s goals and intentions. For example, the intelligent assistant Cortana recognizes when people make promises to others in email and reminds people about their commitments when they might otherwise forget them.

In order for these AI systems to work well, they need to better understand people. The data collected from Ada will help McDuff improve his system and, he hopes, spark a dialogue about the appropriate use of these technologies.

“If I try to put myself in the space of an outsider, it often feels like tech companies are trying to hide the data that they measure about you,” said McDuff. “This installation is intentionally broadcasting that data, and that’s going to raise questions that people don’t always ask.”

Daniel McDuff, a principal researcher in Microsoft’s research organization, checks out the installation of Ada. McDuff built a sensing platform that collects data that is correlated with sentiment. AI algorithms translate the data into a dance of color and light that are displayed on Ada. Photo by John Brecher for Microsoft.

Working across disciplinary boundaries

Inquisitiveness carries weight with Sabin, who in addition to running a private practice in downtown Ithaca is a professor of architecture at Cornell University, where she encourages her students to take a different approach to architectural design.

“We often think of the architect as drawing the napkin sketch, what we call the parti diagram, and that becomes the big idea for the project. Whereas I’m saying let’s flip that upside down and think about how that final form emerges from a set of relationships,” she said while giving a tour of her lab in the basement of Cornell’s architecture school.

The shelves of Sabin’s lab are lined with prototypes from her unique approach, such as a progression of airy, porous 3D printed bricks informed by her collaborations with biologists and engineers that is leading to a reimagining of walls. A lone nylon sleeve knitted with photoluminescent yarns hangs from a vent and hints at her research on the interplay of light and perception that’s manifested in Ada.

Assorted 3D printers, a laser cutter, kiln and a robotic arm more familiar to an auto-assembly line reveal Sabin’s deep dive into cutting-edge fabrication techniques that could transform how architectural designs take shape in the world. Ada’s cells and cones, for example, are created with a process called 3D digital knitting.

In addition to using tools from other disciplines, Sabin teaches her students how to leverage and apply ideas and methods from other scientific areas. She’s spent 14 years working with cell biologists, materials scientists and mechanical engineers to innovate materials and architectural designs that are adaptive, immersive and responsive to the environment and the people they shelter.

“How do we personalize our spaces? How do we start to think about architecture as more fundamentally human and personal as well as sustainable, functional and performative?” said Sabin.

Emergence of Ada

The scene at Jenny Sabin Studio across town is more cerebral, less maker space. Designers click digital models of Ada displayed on computer screens. Glass bookcase shelves are heavy with commemorative statues and plaques. A framed newspaper article about Lumen, an outdoor installation at MoMA PS1 in Queens, New York, that informed the design of Ada, hangs behind Sabin’s tidy desk.

Sabin and her team perform the design work for commissioned projects and exhibition pieces in the studio.

“I’m fundamentally interested in how the research impacts architecture, not just the teaching and the thinking, but how it impacts living architecture,” she said.

On a cold March day earlier this year, near-final digital renderings of Ada showed how the structure would fill the airy nook in the atrium of building 99 and translate data into color and light.

Sabin named the installation after Ada Lovelace, a 19th century polymath. In 1843, Lovelace proposed using punch cards to solve mathematical equations on the inventor Charles Babbage’s Analytical Engine, a never-built forerunner to the digital computer. At the time, punch cards were being used to program Joseph Jacquard’s mechanized loom to weave complex textiles such as tapestries.

“Ada is often credited with being the first computer programmer,” said Sabin. Naming the installation in her honor is especially fitting, Sabin added, because the project brings fabrication technologies such as 3D digital knitting to the visualization of data that powers AI.

Jonathan Lester, left, principal electrical engineer at Microsoft’s research lab in Redmond, Washington, and Asta Roseway, right, a principal research designer who runs the Artist in Residence Program, prepare LED lights to embed in Ada’s fabric. Photo by John Brecher for Microsoft.

Architecture meets AI

For more than three weeks this summer in building 99, Sabin, Roseway and their teams of researchers, designers and engineers connected nodes and rods into the exoskeleton, stretched the fabric into place, strung LEDs, mounted stage lights and hung the tensegrity cone.

Meanwhile, McDuff’s sensors began collecting data in public spaces throughout the building – the atrium, office kitchens and common areas.

Each sensor is the size of a hide-a-key box and consists of a web camera and microphone wired to a black box the size of a cable modem that sends data over Wi-Fi to a secure Azure database.

“If I am complaining about something and I’m angry and everything is going wrong, then it will hopefully detect that as negative, and if I am happy and it is sunny outside and positive, then it will detect that,” explained McDuff.

Horvitz, the director of Microsoft’s research organization, noted his teams have already been exploring the value of harnessing AI technologies as part of building operations. For example, in 2012, his AI team developed and integrated into building 99 a system that employs machine learning and sensing to proactively order elevators based on observations of the patterns of people moving through the building. The system remains a part of daily life in the building.

Separately, Schneider Electric, a global company working to digitally transform energy management in homes, buildings and industry, is working with Microsoft researchers to test whether AI could help reduce the carbon footprint of HVAC systems that are used to heat and cool large buildings.

Beyond applications of real-time sensing and action, AI is also starting to play a more fundamental role in design and engineering including methods that are helping architects explore out-of-the-box design possibilities under the real-world constraints of shape, strength and utility, noted Horvitz.

“One can imagine other uses,” he added. “Like, how might environments in the future shift to make themselves be more conducive to collaboration based on the participants, and their goals and needs?”

Ada’s algorithms are designed to convert any source of data into color and light. For example, the researchers imagine the smart canvas could also be used to visualize music, building vibrations, the number of people in a given space and other yet-to-be imagined scenarios.

“Anybody in here,” Roseway said with an arm gesture sweeping across building 99, “can plug a mod in and get the piece to respond.”

For now, McDuff’s sensing platform controls Ada’s responsive light network. His final sensor is attached to the base of the tensegrity cone, allowing people within the pavilion to drive Ada. Standing there, looking around in awe, smiles come easy. The goal is for Ada to respond in kind.

Top image: Ada, designed and built by Microsoft artist in residence Jenny Sabin, fills an airy nook in building 99 on Microsoft’s Redmond, Washington, campus. The installation translates data into color and light. Photo by John Brecher for Microsoft.


John Roach writes about Microsoft research and innovation. Follow him on Twitter.