REWIND’s high-flying work with Microsoft HoloLens

“HoloLens can solve a whole host of problems. It’s world-changing,” says Sol Rogers as he puts one of Microsoft’s mixed-reality headsets on me.

A hatch opens in the table in front of me and a hologram of a single-seater raceplane rises up. Propellers already spinning, it picks up enough speed to hurtle forwards before disappearing into the air. Later, I see it flying through a hologram Air Gate (two, large inflatable cones) and zig-zagging through another three air-filled pylons. The whole circuit is then laid out on the table in front of me, the wireframe outline of the racetrack filling in with colour and satellite data, bringing the race to life.

Another plane is introduced to the race, flying within inches of the first, their wings almost touching. They battle for position through every twist and turn, until the first plane crosses the finishing line a split-second ahead of its rival.

HoloLens is revolutionising the Red Bull Air Race World Championship. In these races, pilots fly their raceplanes at speeds of up to 230mph just 50 feet off the ground, so it would be too dangerous to have more than one aircraft on the track. This means that fans who travel to watch the race essentially see a series of time trials, rather than planes going head-to-head, making it hard to judge the performance of one raceplane over another. Fans viewing the race at home on TV can see the Ghost Plane that can be used as a benchmark; those at the track can’t.

HoloLens can solve that problem – fans will be able to look at a TV to see a live feed of a plane racing against the clock and also view a hologram of the same raceplane and how it is faring against its rivals in real-time.

Flight Deck, as REWIND’s Red Bull Air Race experience is known, offers a glimpse of this future. While the live feed is something the REWIND team are working towards, Flight Deck can currently teach spectators the rules of the motorsport and how the raceplanes work, with fans able to stand in the same room as a full-size hologram replica of the aircraft. Other experiences planned for the future include holograms of hangars and Race Airports, so fans can see where the planes are kept.

“Very few people have sat in a raceplane, so almost no one understands how big they are, what size they are, what makes them different from other planes,” says Rogers, Founder and Chief Executive of REWIND.

Flight Deck was launched in the VIP lounge at the San Diego race last year, and has since been rolled out to the other seven competitions in the series, which is held across the world every year. An app is also on the way before 2019.

The races see 14 of the world’s best pilots navigate an aerial racetrack featuring 25-metre-high, air-filled pylons in the fastest possible time, incurring as few penalties as possible. Competitors win points at each race, and the pilot with the most points after the last race of the season becomes the Red Bull Air Race World Champion.

“We make a raceplane appear on the table and walk spectators through the key aspects of it: how the tail works, the power, the motor. It gives them some useful additional knowledge,” Rogers adds. “We then make that plane take off and fly away. Next, we lay out a racetrack in front of them and explain the rules of the sport in a really cool, innovative way, to get them up to speed quickly, because most people have never been to an air race. Then we show them two planes that race around the table in front of them. It gives live spectators the same experience as the people watching it at home on TV.”

None of this would be possible without HoloLens. Rather than put users in a fully computer-generated world, as virtual reality does, HoloLens allows users to place 3D digital models in the room alongside them. As the Windows-10-based product does not have wires or external cameras, or require a phone or PC connection, users can walk around the objects they create and interact with them using gestures, gaze and voice.

It has been used by NASA to recreate Mars in its offices, allowing scientists to virtually conduct operations on the Red Planet. Audi, meanwhile, is using HoloLens to communicate with customers and help with engineering; while Saab has called it “transformational”.

Leila Martine, Director of Product Marketing at Microsoft, sees first-hand the excitement HoloLens is causing. “HoloLens is helping companies to work better by empowering staff. Every day we are seeing that workers from a range of sectors can easily collaborate to make complicated problems simple to solve. It really is taking human experiences to the next level.”

Virtual, augmented and mixed reality is becoming increasingly important to companies across the globe. According to market intelligence firm IDC, “worldwide revenues for the augmented reality and virtual reality market will grow … to more than $162 billion in 2020″.

REWIND is at the cutting-edge of that market. The company, which is based in St Albans (Rogers: “We’re only 2.5 miles outside the M25, so we’re London”), was only founded in 2011 but has grown quickly, boasting a team of more than 50 people. The group has already created a multi-award-winning virtual reality spacewalk for the BBC, as well as experiences with Jaguar, Lexus, Nissan, Rolls-Royce, Nike, Stella Artois, Savills and singer Bjork, among many others.

That level of technical experience led to REWIND being added to Microsoft’s HoloLens Agency Readiness Partner programme, which means the company will help other businesses use the mixed-reality headset to transform how they work. Rogers is excited by the possibilities.

“HoloLens is the first device humans have ever had that can augment human intelligence in real time. We have the world’s knowledge at our fingertips with one of these [he holds up his smartphone] but it’s a layer away, a search algorithm away. We have laptops, but what if the second screen is a HoloLens screen? If I can make this [he points to my laptop] as good as talking like we are now, as though I’m really here in the same space as you [when I’m really somewhere else], then why do we need to commute in the way we currently do, why do we all need to be compacted down this end of the country? What if you don’t like the weather so you change it to something else? That’s a little far away, but it’s not that big a leap. HoloLens has some amazing stuff, which is just the tip of the iceberg of what mixed reality can do.”

However, rather than see what he can do with HoloLens in the commercial sector, where the device has been predominantly used since its launch in 2016, Rogers wants members of the public to get their hands on the technology, too.

He was addressing an audience of journalists at Microsoft’s headquarters in Redmond in January 2015. “We are constantly processing terabytes of information from all around us. Imagine if technology has the same sensing capability to process all of this input about the world around us, so it can see the world as we see it.”

Rogers had his solution.

“[With Hololens] you’re adding objects into the real world, instead of creating the entire world as well, so it’s pretty straightforward,” he says. “Inside the Red Bull Air Race experience are the planes, the pylons and a few bits and pieces, so the geometry wasn’t that difficult to make; whereas in the VR version we had to make the raceplanes, the pylons and everything else, and then make the world as well – trees, buildings etc. That takes a lot of time. With HoloLens, we built everything in 12 weeks. That’s pretty quick.”

Three months is very efficient when you consider the experience is used by VIPs at world-famous events attended by tens of thousands of people. In the run-up to that first race in San Diego, the clock was ticking down and the pressure was on; REWIND knew they had to get it right and make sure they weren’t creating something in the “barfogenic zone”. The term was coined by Thomas Piantanida, principal scientist at SRI International’s Virtual Perception Program, in the early 1990s when there was a boom in companies believing (mostly incorrectly) that they could create virtual reality headsets and experiences – virtual reality roller-coasters were a particularly popular, and bad, idea. Piantanida stated that the human brain can cope with slow, jerky frame speeds and faster live action of around 30 frames per second. In between these two speeds – from four to 12 frames per second – is the “barfogenic zone”, where there is a vomit-inducing discrepancy between what the brain expects will happen and what it actually processes.

“The number of people who say virtual makes them feel sick to this day…” Rogers says with a sigh. “It’s not the virtual reality, it’s bad content on bad hardware. You need to understand the difference between what the technology is doing and what the content creators are doing. You can go to the cinema and watch a terrible B-movie but you don’t blame the cinema, you blame the people who made the film.

“We want good stuff to be seen by everybody, especially in new technology. [Joining the Mixed reality Partner Program], it’s nice to be told we are doing work in the right way by the people who made the hardware. To get that approval means that when we talk to new clients they know they can trust us to deliver for that device.”

The success of Pokemon Go has brought mixed reality devices into the mainstream – at one point the game was believed to have attracted around 21 million daily active users. Spurred on by Pokemon Go’s success, other companies have jumped on the technology – Epic Games has Project Raven and Acer has developed its own headset, among others.

REWIND has stayed ahead of the pack in this complex world of coding and technology by doing something rather simple: hiring people who think differently.

The St Albans office is home to people with dyslexia, dyspraxia, ADHD, colour blindness, the list goes on. Rogers has “life-coached” them for years, ever since he taught animation and visual effects at the University of Hertfordshire. A succession of very shy and scared teenagers walked into his classroom after experiencing a “terrible time” in the UK education system, Rogers says. “Schools do not allow those sorts of people to flourish. You have to fit with a very regimented way of learning, being judged and ticked all the way through.”

What he did was tell them: “You are going to make computer games and visual effects, experiment with technology, you’re going to be great, so let’s do this together.”

According to Rogers, it turned out to be the “best interview process” for his future career at REWIND. He had a group of hard-working, bright, young workers who thought outside the box.

“In the world that we are in now and the type of work we create, you need art and logic, you need the left and the right side of the brain working in conjunction. Most people in society don’t have that, you are either one or the other. These are the only people who really do have their brain operating in a slightly different way. Many would see that as a problem or a disability, whereas we see it as a real opportunity. All of them have what I call ‘superpowers’.”

Different is good in the REWIND office. Those people will see a different way of doing something when everyone else is left scratching their heads. Rogers says he feels “lucky” to have found such a talented group of people, many of whom have joined him on his journey from teacher to boss.

Rogers worked for Sony for a few years, making computer games, before entering the classroom for a further 15. Inevitably, he got itchy feet and thought he would set up his own company – “how hard could it be? Pretty hard, as it turns out.”

He was one of just a handful of people who set up REWIND just a stone’s throw from their current office. They outgrew that space in three years, with the team swelling to around 30 people. They now have 50, and an office in San Francisco.

Oliver Kibblewhite, Head of Special Projects, remember the company’s long history with Microsoft. “One of my first jobs was running a Kinect-driven, interactive, digital billboard in central Berlin for the launch of Forza for the Xbox One. We also did some virtual reality projections on the outside of Microsoft’s headquarters in Berlin for the launch of that console.”

Rogers’ face lights up. “Oh yeah, I remember that. We had to close down a whole roundabout,” he says.

They quickly became experts in producing content for virtual reality, augmented reality and mixed reality devices. The level of demand for their services is so high that they have to turn away work; because HoloLens is relatively new, there aren’t enough people who can develop high-quality programs for it.

Rogers says a lot of his clients don’t really know what they want when they walk in the door. “A lot of people come to us and say: ‘We would like you to make us one of these, this is all we know’. And we say’ ‘well, that’s kind of interesting but have you seen one of these?’ And they say: ‘Oh, I haven’t seen one of those’. That might be HoloLens.

“People come to us thinking they need a VR solution and we show them what mixed reality can do; that’s beginning to turn the tide, people are saying, ‘actually, I think that might be more fundamental for us’.”

The experience is all that matters for the REWIND team. Their office reportedly littered with holograms of dogs and fishes with alien heads that spring to life when you put on a HoloLens. At his home, Rogers has a hologram of a galaxy that’s floating around in his bedroom, just waiting to reactivate itself when he uses Microsoft’s device.

“The memories we build from using HoloLens make us feel the same way as memories of the real world. The physicality of those objects and all those experiences become intrinsically connected with our real world in a way we haven’t had with anything else before. They stay in your memory in a way that, if I turn away from you, my brain remembers that you exist because you’re still in the room with me. HoloLens does that with a digital layer. We are really excited about the next steps of mixed-reality.”