‘Unmoored’: Times Square installation shows how artists can anchor storytelling with mixed reality
Athima Chansanchai
For three days in July, Times Square will be underwater.
During that time, visitors to one of the most famous landmarks in New York City will experience the usual sensory overload of the bustling area, matched by another one floating 26 ft. above – a jam of more than a hundred ocean vessels. Time will accelerate and spawn clusters of anxious plankton. Boats from the New York waters, essential for commerce and leisure, comingle with tiny organisms that are essential for life on earth.
But the visitors will stay dry.
The experience results from the unique marriage of mixed reality technology and art called “Unmoored” by Mel Chin, one of many exhibits around the city as part of “Mel Chin: All Over the Place,” a comprehensive survey on four decades of Chin’s work co-produced by the Queens Museum and No Longer Empty.
For Chin, the piece answers a question he asked as soon as he saw Microsoft HoloLens, which combines digital and physical worlds through a head-mounted device: “How can we create a scenario that projects a presence of the future? I can tell you the story about what I see but mixed reality allows you actually to see and hear it.”
His storytelling envisions one possible future of New York City if climate change continues unchecked: Times Square suffering a reality due to melting ice caps, erosion and coastal flooding.
“The future of our world is important,” Chin says. “We hear scientists talking about a distant future – of a world under 30 feet of water. It’s probably not something that you will experience within your lifetime unless something catastrophic happens. This project intends to have its own form of drama and allows people to absorb and meditate on what it would be like. It should be a powerful digital expression. So, if it is the future, how can such a prognostication make us more aware of the present?”
Mixed reality can empower an artist with a tool that brings unique visions to life.
“Artists have a really unique ability to tell stories, and they also have a really unique ability to see the potential of any new medium or any new technology platform,” says Lorraine Bardeen, general manager for the Mixed Reality Workplace at Microsoft. She has spoken extensively about using mixed reality as a vehicle for storytelling, including sample apps like “Actiongram” that give HoloLens users the power to insert holographic characters into their stories.
It’s also significant, Bardeen says, that storytellers are able to take any type of digital content – visual, spatial and audio – and place it anywhere in the real world in a way that people can discover it, accidentally or intentionally.
“We really believe that the true value of mixed reality comes from both being in the real world and bringing more value where you need it right at the moment you need it,” she says. “It brings us to the part of our childhood when we understood that there was a lot more in the world than we could see. Usually when children see holograms integrated with the real world, they’re zero percent amazed. So there’s something there about getting to a human core that we’ve maybe pruned off that I think artists are uniquely able to recapture.”
To help people get back to that core, they can check out the mixed reality piece in Times Square, which is paired with a physical work from Chin, also at the site, called “Wake,” commissioned by Times Square Arts. “Wake” is a sculpture based on a 19th-century shipwreck, USS Nightingale, merged with the skeletal remains of a marine mammal. The vessel’s figurehead is based on 19th century opera singer Jenny Lind, who Chin calls “the first superstar of America.”
The exhibits are very purposefully set in Times Square, which Chin says is “immersive in its own level, full of the marketing of superstardom.”
Chin’s juxtaposition of the past, present and future creates an inter-related experience that begins with what Chin describes as the beginnings of mass marketing in “Wake.” That industrial and star-studded past ties it to the present-day Times Square and its part in New York City’s 19th century expansion. And then “Unmoored” takes visitors to the future.
“When we critically examine the history of international trade and mass production of raw materials like iron, oil, steel, lead and coal that came out of the 19th century industrial revolution, we understand the reality we are facing now,” Chin says. “
The project evolved from a partnership with teams from Microsoft and Listen (a company that often works with Microsoft and other brands on an array of artistic experiences), brought together by Zengalt, a Bellevue, Washington-based group of technology enthusiasts who specialize in mixed reality technology and other similar forms of technology.
The idea came from a meeting at another exhibition at the Peterson Museum in Los Angeles: “An American Supercar: Experiencing the History of the Ford GT in Mixed Reality.”
“So we all got together and using Microsoft’s technology, the wonderful work that Listen does with their audio technology and with our development services, and we looked for ways to bring this technology to an even wider audience,” says Billy Watenpaugh, who runs business development for Zengalt. “It’s a nice challenge and something that really allowed us to pioneer this type of storytelling in a mass market. This is the biggest thing that I’ve seen or heard being done on this scale in this type of environment.”
He calls out the interactive aspect of the technology, which is different for every person wearing the device, so that what you’re seeing is particular to your motions, interactions and gaze. But there’s also a communal understanding.
“Mixed reality is fascinating technology because it allows friends and family to experience the same story together in the same environment,” Watenpaugh says.
“The immersiveness of the HoloLens experience is something that we were all really attracted to, and there really isn’t anything else that provides a similar experience,” adds Sarah Ibrahim, the technical lead for both the creative development process and the physical activation.
“Unmoored is about the city and its identity unmooring from its current state and becoming something else,” she says. “I think that mixed reality is a very interesting platform for public art and I think artists have limited access to these types of technologies, so we can keep creating a conduit for artists to access it. If we can provide opportunities for artists to continue to create augmented reality public art, think about how much more art there can be with the scale of augmented reality.”
Ibrahim says the process of creating a piece like this involves many layers and many different kinds of skills.
Massive teams of developers, animators, 3D artists and sound designers worked collaboratively with Chin, who had his own team at the University of North Carolina Asheville (where the former New Yorker currently lives) research the boats moored in New York Harbor, which would then appear in the augmented reality of “Unmoored” above visitors’ heads. Chin and his team produced a comprehensive list of 141 boats (calculated to fit the space above Times Square) and information such as models, years, draft height, lengths and number of motors. Some are more than a hundred feet long; others are rowboats.
“As an artist, I can dream and postulate many things, but it’s unlikely I can bring all of them to reality working alone. This is an opportunity to realize a vision comes by working with teams from Microsoft and Listen who can do what I cannot do,” says Chin. “That’s the possibility when you have a collaboration between art and modern technology. I’m thankful to all the crews that made these projects happen, all the people who have entrusted me to do something, we hope is, extraordinary.”
Each boat has a unique number that can be looked up on the piece’s microsite, which will provide links to scholarly articles or research about climate change topics.
It’s what Chin calls “a confirming flood of science that the imagination senses.”
But in order to reach more people, and because the HoloLens experience in Times Square will only last three days with a limited amount of devices, the teams have also created a version of “Unmoored” that will be available on smartphones.
“As an artist you can make a statement that this is accessible to everyone who has this type of phone, of which there are many. It’s more like a portal,” Bardeen says. “You get a peek into this way of thinking about the world. And so sometimes a portal is just what you need to pique people’s interest.”
Chin sees the project as an opportunity to “compel curiosity, to reanimate desire to understand, to reinvestigate what you don’t know in another way.” He hopes it ignites a sense of exploration for visitors – who can number 300,000 every day during the summer – and that they think about what they saw, and investigate on their own later.
“I think it’s a challenge for all of us working in the creative field to find ways that can move people into options that don’t exist yet,” Chin says. “And quite simply ‘Unmoored’ and ‘Wake’ are pieces that deliver options to be considered. And that’s the best we can do.”
