Mixed Reality & AI Zurich Lab – a Lab is born

 |   By Jim Pinkelman, Senior Director, and Scarlet Schwiderski-Grosche, Director Microsoft Research, Cambridge

Alex Open

In the first week of October, the eyes of the European computer vision community were on Zurich where Microsoft launched the Mixed Reality & AI Zurich Lab. The Lab is led by Professor Marc Pollefeys who has a joint appointment at ETH Zurich and Microsoft. 

As Partner Director of Science at Microsoft and Professor of Computer Science at ETH Zurich (Swiss Federal Institute of Technology in Zurich)Pollefeys and his teams benefit from research collaborations between the two organizations in his research domain, Mixed Reality (MR) and Artificial Intelligence (AI). According to Pollefeys, this arrangement is “a real win-win between what ETH and what Microsoft can offer”. At ETH, which is ranked second in the Times Higher Education World University Rankings in Computer Science 2019, Pollefeys works with top students for his research. 

Lab Team
The Cognition Zurich team with Alex Kipman in the new Lab.

At Microsoft he and his team are part of Alex Kipman’s Cognition organization that leads Microsoft’s effort in MR and AI including Kinect and HoloLens products such as the recently announced HoloLens 2. “In this way we can provide great opportunities for students to work with Microsoft, to get access to devices and resources that extend beyond what they have at ETH. There are a lot of open research problems for computer science students to work on with a clear and exciting career path”, says Pollefeys. 

Before starting his joint appointment at Microsoft and ETH in summer 2018, Pollefeys had spent two years at Microsoft headquarters in Redmond (USA), working full-time directly under Alex Kipman on HoloLens technology while on leave from his ETH faculty position that he had held since 2007. According to Pollefeys understanding all sides, academia, industry, as well as the local Zurich context, well in my research field has given me a vision of what I want to achieve with my work, namely building the future of mixed reality using computer vision to map and understand the environment, recognize and track relevant objects, and assist users performing tasks. We are also exploring the synergies between mixed reality and robotics. 

Investment in and for Switzerland

The Mixed Reality & AI Zurich Lab was announced during Microsoft CEO Satya Nadella’s visit to Zurich in October 2018. Nadella had commented on the potential of the new Microsoft research lab in Zurich, especially access to world-class talent whose research could contribute to Microsoft product development as well as advance the science in the fieldBut he was also curious about any collaboration between the Lab and local industry for exploration of HoloLens and other MR technology in their application scenarios. Microsoft technology should make the whole spectrum of SMEs (Small and Medium-sized Enterprises) to multinationals more competitive and the public sector more efficient suggested Nadella at the time

The Lab that enjoys close ties to the local Microsoft subsidiary is in an ideal position to make strong contributions especially for Microsoft Switzerland customers. Marianne Janik, Country GM of Microsoft Switzerland commented: “I am very pleased that we can further expand our presence in Switzerland with Marc and his team and that we can bring the close cooperation with ETH, which has existed for over 10 years, to the next level. With the Mixed Reality & AI Zurich Lab, we can not only support basic research at ETH Zurich, but also support Swiss organizations in the development of innovative products and services.” 

Close collaboration with ETH Zurich

Over the past year, the Lab has almost grown to its initial target size of 1full-time employees and is located in a new space in the heart of Zurich city center, less than 15min from Pollefeys’ ETH Lab – the proximity allowing frequent interaction and ongoing collaborationPollefeys splits his time between his ETH and Microsoft offices and supervises research teams at both. Besides close collaboration between the respective teams, the Microsoft and ETH researchers participate actively in the wider academic ecosystem and use the Swiss Joint Research Center and Inria Joint Center to drive open and collaborative research projects with key academic collaborators at ETH, EPFL, and Inria in France.   

Various Lab opening activities

The Lab’s launch event took place from 2nd to 4th October in Zurich. For the opening, many Microsoft experts traveled to Switzerland, including special guest Alex Kipman, the inventor of HoloLens and Kinect, as well as Microsoft Research (MSR) Cambridge Lab DirectorChristopher Bishopand MSR Redmond Lab Director, Donald KossmannSeveral prominent computer vision researchers from leading European research institutions and from Microsoft Research and Cognition also honored us with their participation. 

One highlight of the launch was Alex Kipmans public talk at ETHAlex gave an ETH Global Lecture entitled “HoloLens 2 – Unpacked” where he described the engineering and research accomplishments that went into the HoloLens 2 to approximately 400 ETH students and faculty in ETH’s main lecture theatre, the Audimax. The audience’s excitement was palpable, and many attendants stayed on to talk to Alex and the Cognition team during a subsequent drinks’ reception. The recording is accessible on the ETH event page.  

Alex Kipman
Alex Kipman kicking off the ETH Global Lecture (photo: ETH Zurich/Andreas Eggenberger).

The main component of the launch was a scientific workshop with talks from Microsoft researchers and ETH Zurich and other academic collaborators, especially from our the Swiss Joint Research Center and Inria Joint Center. There were considerably more attendants than expected. Of the 110 attendantsabout a quarter were from Microsoft, half from ETH Zurich, and others joined from EPFL, HWTK Leipzig, Inria, MPI, TU Munich, and University of Zurich. The agenda was a mix of presentations from Microsoft and academic collaborators. For more details, please check the workshop event page.

Group photo
Workshop attendants in the historic workshop venue.

In addition, two evening receptions were held in the new Lab, the first was organized jointly with Microsoft Switzerland and the second was part of the workshop. On 2nd October, the Microsoft Switzerland team had organized a VIP Reception with invitees from Swiss industry, politics, and academia. Among the guests were senior representatives from companies such as ABB, Balgrist University Hospital, ESRI, Hilti, Omya, Schindler, Stadler Rail, and Zuhlke as well as from ETH Zurich. There were demos to the visitors. Microsoft Switzerland Country Lead Marianne Janik and Pollefeys gave short speeches. 

Lab opening
Microsoft Switzerland GM Marianne Janik with Marc Pollefeys and ETH President Joël Mesot (right).

The Lab’s research journey has journey has only just begun. While research can often lead in unexpected directions, the three research themes at the outset are:

  1. Crowd mapping to allow sharing and persistence of Holograms at world scale. This work directly links to Azure Spatial Anchors, but also looks at longer term challenges such as privacy preserving image-based localization methods.   
  2. Spatial intelligence for autonomous robots and human-robot interaction. 
  3. First-person activity recognition.

At the end of an exciting week, Pollefeys acclaimed “It has been a great start and I could not be more excited about the research journey upon which we have embarked in Zurich. We have an incredible group of people full of ideas and passion, and I am extraordinarily optimistic about what we can achieve here. 

Interested in more information? Please visit the following links:

The title picture of Alex Kipman was also made by Andreas Eggenberger on behalf of the ETH.

Tags: , , , ,