Skip to Main Content

World premiere uses Kinect to conduct, puts audience ‘inside the instrument’

The maestro raises his right arm with an abrupt, exacting motion. Then he raises it higher and higher still, slicing the air. His left hand juts out in a fist that remains clenched. With an avalanche of notes, the piano comes to life.

This is what it looks like to conduct with Kinect.

Seattle Symphony Music Director Ludovic Morlot used these gestures earlier this month to control a set of reed horns, chimes and a grand piano equipped with a Kinect for Windows and robotic devices during the performance of “Above, Below and In Between,” by Seattle-based kinetic sculptor Trimpin.

It was the world premiere and only performance of the 20-minute piece, which also included a soprano and nine human musicians — a collaboration two years in the making.

The audience was enchanted.

There were giggles of delight as Morlot’s raised right hand corresponded with an increase in volume from the piano, and gasps of surprise when the chimes wailed in surround-sound cacophony. In addition to the Kinect making the evening unusual, the concert took place in the symphony lobby, a grand space three-stories high, with lights from nearby office buildings twinkling through the floor-to-ceiling glass.

The chimes and human-played instruments surrounded the audience on the mezzanine. The idea, says Trimpin, is to put the audience inside the instrument. “Music and technology has a long history, going back several centuries,” he adds.

Seattle Symphony Music Director Ludovic Morlot practicing the gestures that control the kinetic instruments during a dress rehearsal for “Above, Below and In Between.”
Seattle Symphony Music Director Ludovic Morlot practicing the gestures that control the kinetic instruments during a dress rehearsal for “Above, Below and In Between.”

The piano is equipped with a plunger-like structure over every key that plays the notes, sometimes at breathtaking speed, based on a computer program. The conductor stands behind the piano, facing the audience, a non-traditional view that gives those watching a front-row seat to his gestures.

Morlot, who has two young daughters, says he’s familiar with Kinect as it relates to games on the Xbox. (He likes to play tennis and soccer.) But when it comes to conducting, it’s a new skillset. “It’s taken me out of my comfort zone. I’ve been old-fashioned in a way, studying music. The element of bringing technology to it is foreign to me,” he says. “That’s what attracted me to it.”

Dimitri Diakopoulos, who works in the Bay Area doing interactive design and computer vision research, wrote the code for the piece and created the custom gestures. He says he choose the Kinect because the body-tracking algorithm in the Kinect SDK is predictable and reliable, given its wide adoption in a variety of creative and interactive environments.

Interactive designer Dimitri Diakopoulos says he choose Kinect for Windows because the body-tracking algorithm is reliable.
Interactive designer Dimitri Diakopoulos says he choose Kinect for Windows because the body-tracking algorithm is reliable.

Microsoft is hoping to encourage more creatives to use this technology and experiment, over the next few months we will share more stories of creators.

“Conducting is a rich language full of subtlety, nuance and vigor. I imagined it would be an interesting challenge making Trimpin’s mechanical instruments understand a tiny bit of this language,” Diakopoulos says.

The biggest challenge? Light.

“Light can be a killer,” Diakopoulos explains. During the performance, the Kinect sensor was outfitted with a black backdrop. A PC running behind the scenes was hardwired to the kinetic instruments. But the Kinect itself and the tablet that gave the maestro visual feedback for what he was controlling both operated wirelessly.

Seattle-based kinetic sculptor Trimpin calls the piece a journey that puts the audience inside the instrument.
Seattle-based kinetic sculptor Trimpin calls the piece a journey that puts the audience inside the instrument.

Diakopoulos says he tried to fashion the custom gestures after Morlot’s natural movements when conducting the human-played symphony. But there were some notable differences. First, when the maestro summons the traditional symphony to begin, he raises his hands, then lets them down. For the Kinect symphony, Morlot kept his left hand a fist and the other raised while the instruments played on.

Trimpin, who is currently an artist-in-residence at the Seattle Symphony, calls the piece an “experience,” a journey to be watched as well as listened to. Part of the Symphony’s “[untitled]” series of concerts, it gives Morlot the opportunity to make room for music that typically would not be programmed into the regular season. With a start time of 10 p.m. and a more casual atmosphere, the idea is to also attract a non-traditional, younger audience.

Diakopoulos was pleased by the audience’s enthusiastic reaction, particularly because they had no idea what to expect.

The world premiere was the one and only performance of this Kinect-conducted piece.
The world premiere was the one and only performance of this Kinect-conducted piece.

Adds Trimpin: “We’re already talking about how to extend this idea, how to bring it to the next level.”

Morlot says he’d like to conduct an entire symphony piece using Kinect someday. He’d also like to rig the instruments so that he can control tempo, as well as the volume, using gestures.

The fact that the show was the first and only performance of the piece adds magic to it, he explains. “There is an adrenaline for the audience and the musicians of only going through that journey once.”

Though the performance was only a one-time thing, the installation will be open in the Seattle Symphony lobby for the public to view Wednesday afternoons in June from 12 to 2 p.m. Also, local student composers will hold a concert using Kinect to conduct Trimpin’s instruments June 1.


Photo credits: Jason Brown