Craig Mundie: College Tour – New York University

CRAIG MUNDIE: Thank you, and thank you for coming and joining me this afternoon for what will hopefully be a good dialogue.

Speaking at New York University, Craig Mundie, chief research and strategy officer, Microsoft Corp., announced the Games for Learning Institute (G4LI), a partnership with NYU and other universities to investigate which qualities of computer games engage students and develop relevant, personalized teaching strategies. Oct. 7, 2008, New York.

I have a very cool job at Microsoft. I’ve been there 15 years. I went there originally to think about how we would put computing into just about everything, and in 1992 that was a fairly novel idea, but most of us are starting to live a lifestyle where we, in fact, have computing in everything.

In July this year, Bill Gates retired from daily operations at Microsoft, and two years ago, we split his job in half. I took half and Ray Ozzie took half.

I feel lucky to have the half I got in that I’m responsible for the global research operations of Microsoft, which is about a thousand PhDs in computer science and many other subjects these days that we think all come together to form the future of computing.

I also have responsibility for a lot of our geopolitical activities on a global basis. I’ve led a lot of our activities the last 10 years in building our company’s activities in China, India, Russia, Indonesia and other places that are increasingly important on the global stage and in the world economy.

But I also am fundamentally a startup guy at heart, and that’s why I went there, to Microsoft, in 1992, and since that time I continue to do startups. I have three that I’m doing inside the company now, one in health care, one in education and one in making products for the emerging middle class on a global basis.

So, the combination of leading-edge technology and the ability to start these new things is something I relish and will continue to do for the foreseeable future.

Throughout that time, I always spend some considerable effort each year trying to talk to people, not only in the academic environment but broadly, and I’ve spent a lot of my time in the last few years meeting with university people and college students outside the United States. Bill Gates tended to focus on some of the trips here. With his retirement, I rebalanced that and I’m sort of taking up the mantle from Bill in maintaining that dialogue with academia on an annual basis.

So, this year I’m visiting five schools in five days all across the country, and hope to learn from you what’s on your mind, but also share with you a little bit about what is going on at Microsoft.

Many people in the world think of Microsoft as, oh, that’s Windows and PCs and I use a word processor, but, in fact, the company is very, very diverse now in terms of the things that we do.

So, I’m going to spend the next half hour and use a few demos to give you an idea of just one narrow area that we’re pursuing around education, which is a topic that we all have an interest in, and then use that to expose some of the underlying technological change that is either happening now or is going to happen in the next few years.

So, this little video clip you saw was Professor Roy Gould, who’s an astrophysicist at Harvard, and it was taped at the TED Conference, the Technology Engineering and Design Conference, where we announced this WorldWide Telescope activity. This had been put together by the Microsoft Research people, and is now available as something people can download and use.

It was the work of many people, but notably Jim Gray, who was a famous Microsoft researcher, who actually died mysteriously in a sailboat accident a couple of years ago. But he had gone out and said to the astrophysicists of the world, you know, we need to unite and we need to find some way to bring all of the tremendous work that’s done there together so more people can use it, we can use it, and more people will be exposed to it.

That led to a big effort to assemble pretty much every known image of all types, whether it was x-ray, infrared or what you might have, optical observations, Hubble Telescope, into one big database.

So, much as you’re probably familiar with the ability to go to Google Earth or Microsoft Live Earth and look at maps or aerial images of the globe, this is sort of turning the telescope the other way around and looking out at space and being able to navigate freely in that environment.

So, I brought today just a version of this, and what you actually see here is Microsoft Surface. This is a computer built into a translucent tabletop, which we announced about two years ago, and it’s basically a touch-based computer system that allows people to have a very natural form of interaction.

So, now what you see on the screen is not the video, but it’s actually this table surface. So, as I just take my hand and move around, I can look at this. If I wanted to zoom in a little bit on this thing, I could do that and move it around. I could push different buttons and select different types of ways of looking at this particular environment.

But I think what I really wanted to share with you as the last part of this particular demo is how it in itself is a multimedia authoring system. Just as you saw, as Roy Gould was talking about, a bit of a tour that was provided by the man who wrote this program, we also found in the development and testing that by building multimedia authoring into it, it allows people to share their discoveries and their interests.

In the last stages of the testing there was a guy named Benjamin who we made this available to, and I wanted to give you one of the tours that Benjamin did. What’s notable is Benjamin, unassisted in doing this, is six years old. So, let’s listen to what Benjamin had to say.

(Video segment.)

(Applause.)

CRAIG MUNDIE: So, why do I show you this? I think it’s a great example of what happens when we give these new types of tools to people, no matter what age they are. But it’s notable to take even fairly smart young six year olds and realize that this kid had a natural interest. His dad and he had been sort of amateur astronomers for a couple of years at that time, and he knew a lot about the sky, but he really didn’t have any way to experience it very directly.

We gave him this WorldWide Telescope and literally in a matter of a couple of hours he had discovered how to use it, to find the things that he had learned about, to integrate it with the knowledge, for example, that he got from the Internet or Wikipedia, and to bring these things together. And in literally a short amount of time he could just push the record button and drive the thing around and explain to you and show you the things that he was in interested in.

I get very excited when I see these things. We have a lot of activities trying to focus on how technology can be used in education. And I thought that I’d basically show you some ideas we have about how that might actually play out as everybody becomes more well-equipped with these types of technologies.

So, this is actually just a Tablet PC that you could go down to the store and buy today, but we’ve rigged it up. I have a group of people who work with me, and we develop prototypes of concepts that we think will be important in the future. It’s a bit like the old story that a picture is worth a thousand words. I could get up here and talk for a long time, show you lots of PowerPoint slides, but perhaps never really convey what we dream about as the way that these things come together to aid education.

So, what you’re looking at is essentially the screen that I’m looking at, just in large scale, and what I’m going to do is take you on a little bit of a tour of what it might be like to be a student at some point in the not-too-distant future where all of these technologies come together, and where the quality of computation and the kind of support that it can provide to you is just qualitatively different than what we enjoy today.

So, let’s say you’re studying a number of courses, one of which might be anatomy, and let’s imagine sort of what the possibilities would be if I can now take multimedia textbooks and 3D modeling and many of these type techniques and use them in lieu of the old traditional concept of a textbook.

In this environment I can do a lot of different things. I can basically not just do mash-ups of the planets or the earth; I can start to do mash-ups of the different layers of physiology. So, I can look at the skeletal system, I can say, OK, show me the muscle system, show me how the nervous system fits underneath all that. And in each of these cases, the computer can be analyzing other kinds of information that might be useful to me, and perhaps come up with a model like this where you can point and click yourself with just using your finger about the things you might be interested in.

So, let’s look, for example, at this one, and it says, OK, this is actually extracted from The Anatomy of the Human Body, which is a textbook by Henry Gray that’s well-known in that environment.

What you can call up is this information. What you’ll see as this goes on that anything that’s outlined and is green is something where there’s been a contextual analysis of the document, and it’s deemed to be highly relevant to the subject that you’re investigating. You get these little cards and down at the bottom you have a sliding scale of cards and other things you can follow to pursue this further.

So, let’s say I want to see a 3D view of this particular anatomy, and I’m not just interested in the hand, I’m interested in going up and doing the head. So, I’m going to basically just touch on this thing and drag it up here to the head, and then be able to zoom in there.

So, now what I’ve really got is a 3D model that I can explore as I see fit. So, let’s just turn this around a little bit and see what the whole brain looks like in there. Then each aspect of this can be annotated, I can take notes, look at it in different resolutions, but I can continue to drill in. So, just like people keep following their nose down Web links, you can essentially follow these graphical metaphors for navigation and say, OK, I’m interested in studying this brain a little bit more.

So, in this environment, sort of the spinning box down there says that there’s essentially some Web Service that’s going on behind this. So, I’m connected to a network and it basically brings forward and says, OK, here’s what’s going on in a brain. You’ve got synapses. They sort of actually look like this under a microscope. It shows now that the scale that’s being presented is 40 nanometers.

And you can say, OK, how do synapses work? So, I again could just follow this link that says, OK, here I can teach you about the details of how a synapse works and how it fires, but I might be able to actually teach you animation. So, I could, for example, select that and it shows essentially how there’s a passage of chemicals basically that go from one part of the synapse to the next, and I could continue to drill in on this environment.

But I think many of the other things that are going to happen as everyone has access to these is some of the things that you enjoy today more for personal communication and entertainment, for example, like social networking, will eventually find their way into being an active component of how people collaborate and share information.

So, for example, I might actually close this and then find that maybe a colleague, a student in the class says, hey, I’ve been studying this too, I see you’re looking at this right now, it’s sort of like a different form of presence, and I can click on that to see what he’s been doing. So, he’s studied the synapse, he’s created his own drawing, he’s annotated it, he’s happy to share that with us, and you can say, okay, that’s useful, it helps me understand a little more.

So, beyond that, we want to look at how the computer can not just be something that reacts to our needs on a momentary basis, but how as the computer becomes more and more powerful it becomes really an assistant to you.

In my job at Microsoft I have a lot of things to do, and I have a lot of people who help me get them done. What’s great about those people is as we work together as a team and have more history, more context, they’re able to do things that anticipate what I might need. They don’t just wait for me to say, hey, go out and do each individual task.

And yet our computers today don’t do that. They don’t try to anticipate what you’re going to know, and yet computing is having a more and more intimate relationship with you and all the tasks and interests that you have.

So, ultimately there’s no reason that we can’t bring these things together to guide an exploration into an educational subject. Today, a lot of that information is being gathered and use for other activities. So, for example, in a targeted advertising environment we want to find all the people who want to buy BMWs this week and send them BMW ads instead of Ford ads.

So, you can derive a lot of information by observing people’s behavior and interests, and yet none of those techniques have really been applied in this academic setting. So, this is another place where we think that that kind of thing could happen.

So, what I really want to do next is say, OK, well, what should I do, how should I proceed in my plan of study? And so here what we have at the bottom is a whole set of types of papers and documents and other things, videos that I could review, and as I go from left to right the color coding goes from highly relevant to more speculative.

So, I can essentially choose whether I really just very straightforwardly want to pursue the subject at hand or whether I want to explore a little off on the edges, and, of course, there’s a huge amount of stuff here. So, I might be able to use filtering, either collaborative filtering or some other type, in order to connect down that.

So, I can basically pick, for example, my study group and say let’s just look at a set of these things that either my study group has already examined or where our collective interests would map very tightly to the subject matter that’s presented.

In this case maybe I’ll decide to just go off and explore a little bit. I’m interested in this caffeine and your brain subject, because I may have trouble staying awake, and so I’ll just click on that one.

So, here somebody comes up, says what the citation is about, a summary of it, but also says, hey, there’s a simulation. If you want to understand how caffeine interacts with the brain or what lobe of the brain that this actually relates to, I can go look at the simulation. So, I’m back to the model. I can basically see an animation that shows where this affects and, in fact, this guy is clever, gave me a little sliding tool that goes from no caffeine up to drinking an energy drink or a cup of coffee, and as I move it I can kind of understand the level of activity that the caffeine might induce.

I think as you look at this you always want to be presented with an opportunity to have the computer offer you more things, and you want to be able to dial in how relevant they are.

So, if you look at these boxes at the bottom left of this thing, as I touch each one of these things and just scroll across them, it basically gives me a shortcut way of looking at the kind of things that’s available. Let’s see, let’s just take the leftmost one and go there.

So, here it’s talking to me about strokes and how the brain is affected by that, and there’s a huge amount of information, and, in fact, some text that’s available about this.

So, here what you want to do is to start to use the combination of the local computing power and the huge resources that exist in the Internet, what now many people call the cloud, and use them to assist in this.
So, I can ask for the whole text of this document to be put into a semantic graph, which allows me to look at the different component parts, and then to have the graph analyzed so that I can understand which parts of the document might be most relevant to the subject matter that I’m interested in.

So, in this environment it says, OK, the stroke scale might be the most relevant part, but you might be interested in magnetic resonance imaging or other stories about brain activity.

So, I’ll say, OK, I want that one about the NIH, and so I can go back and now look at the text.

So, here in a sense the computer has done what many of you might have done if you had read the text and scanned it. You’d highlight the part that you find were the most interesting. So, in fact, it’s highlighted it in colors that correspond to the relevance. I can just touch on one of those paragraphs and zoom in and read it.

In fact, I might be a student who came from Japan and recognized that the original work might have been done in this case at a Japanese research institute, and if I want to actually read it in my native language, I can just click on the button that actually shows me in real time a translation of that back into the original Japanese text.

This essentially gives us the ability to drag and drop things, to cut and paste things just as you would. I say this is actually a pretty interesting one. So, I’m just going to take this text and pick it up and drag it down here into my notebook. When I deposit it there, what I’ve really done is put this into my sort of lifelong personal digital notebook.

A few years ago, we started and built a product now called OneNote; some of you might use it. It’s part of the Office suite. And the idea there was to create a digital notebook of essentially unlimited size that was equally capable of dealing with handwriting or pictures or videos or traditional text, and to just put them in freeform ways into a notebook, but in doing so the computer is then able to analyze them, to index them, to search them, and essentially in real time everything you do becomes part of your own personal searchable history.

So, while this is something we don’t enjoy today, perhaps your kids will grow up and it will have been completely natural for them to have had a digital tablet of some type from their very early ages.

So, since this is in the future, I’m just going to take and sweep this thing all the way back to 8th grade and say, you know, when was the first time I actually had somebody talk to me about some of these brain and spinal activities, and it was maybe a biology class I had back then.

The ability to look at all these things in real time, whether they’re your own pieces of information or the things that come from other people, I think is a tremendously powerful future tool.

So, if I look at this and say, OK, well, let’s go back to the present and we’ll kind of sweep out here to the current time, then what I really am looking at is today. I have a timeline. I’m looking at the fact that while I was just looking at the anatomy coursework, I may be taking art and also literature, and here again we can now correlate across these things, so that the lines can show the semantic relationship of either activities or information that exists from one class to the next, and you can navigate around in this environment.

So, let’s just say that this thing says, OK, today there’s supposed to be a meeting and an upcoming test and a group project, and I’ll talk now a little bit about what collaboration might be like when all of us are armed with these types of digital devices.

So, I’m going to come back here to the table, and in this case we’ve actually coded this device and a number of other ones with a visual barcode. So, just like you have the UPC code on your food packaging, we’ve come up with high information content codes. Since this table is actually based on cameras that look up through the surface and they see and recognize infrared patterns and other things, they’re actually looking up.

So, when I lay this on the table, the little barcode turns out to be a way of just indicating to the table what this device is and what activity it might do. So, it sees the barcode and extracts some documents from it, which I can essentially take and manipulate and turn around as I wish.

But maybe I or one of my other colleagues in the class, they come up with their cell phone where they’ve been accumulating some similar stuff, and so they put it on the table. These essentially just spill out their contents as it relates to the common interests to these people.

Maybe somebody went to the anatomy part of the lab class and actually took a brain sample here, and just a model because you’re trying to understand that, and you say, OK, well, I’ll put it on there and I can turn it around so that I can pull out things that were provided as aids to study with this.

So, in this environment I can essentially move these things around, I can do like I did in the astronomy stuff, I can make them big or small and look at them. I can take them and turn them around and look at anything I want.

And maybe when we’re done, we said, OK, there’s a few of these things that really look interesting for our study group in the class. So, over in the corner here I’ve just got a little folder, and I can just flip things into the folder that I think really are interesting to the rest of the group. And when it’s done, I can just publish this stuff for review.

So, I think what this shows is that there are many ways that people are going to collaborate, and that as all of the knowledge not only becomes searchable but that which you produce and gather has these different ways of being represented, then I think we’re going to see a lot more interesting ways to interact.

One of the big challenges that our global society has these days though is the need to improve the education or health care situation on a global basis, and one of the big challenges is how do you get the infrastructure or this kind of capability in the hands of a lot more people when, in fact, today these are fairly expensive.

So, one of the things that we’ve been looking at are ways of using really novel technologies to do that. I’ll lay this down so you can see it in a second, but this is a thing that’s a fraction of a millimeter thick. It’s actually a flexible display based a bit like the e-Ink technologies that you might see in a Kindle book reader from Amazon. But this one actually operates in color, requires only batteries to operate, and today I’ve just got it with a little driver taped to the back.

But the idea is actually to be able to take a cell phone like this and be able to just have a connector between this and this — or maybe it won’t be a connector, maybe it will be an Ultra -Wideband radio connection. What this becomes is essentially the display for your cell phone.

So, you can basically fold this thing up or just stick it in a sleeve pocket on your notebook or backpack, and when you get someplace where you want to read something or interact with something or potentially do a group discussion about things, or have a videoconference, you just take this thing out and set it down.

I’ll put it here, and you can actually see.

We actually took some of the components of that demonstration I gave you a little while ago, and you’ll see that it’s changing, right not just about once a second or so. It takes a different part of that slide presentation I was just giving you or that demonstration I was giving you, and it basically just cycles it through this thing. But that entire image and all of the transformation of it is all being done on this card, which is completely flexible.

So, we think it’s just another great example of how people are going to be very clever about using advanced technologies to not only provide those of us that have a pretty well-off economic situation access to advanced technologies, but how these things can be very key to solving some of these huge challenges where people don’t have as much infrastructure and don’t have as much disposable income, and yet they have a real deep hunger to be able to gain access to these things.

It’s an area that I’m personally passionate about and one that the company is very focused in. Our research lab in Bangalore, India was giving a specific charge to investigate how to use advanced technologies to help with health care and education in rural environments, and a number of those things are already being deployed. They’ve changed in some ways the way that young children are being taught in some of the rural villages in southern India using simple things like one PC, a DVD burner and a TV camera – a simple Handicam. And in doing that, they were able to create a way of recording the best teachers in each district in each subject.

When they originally did that, it was with the eye toward letting the students in every classroom see the best teacher, but there were some unintended consequences. They gave the DVDs to the teacher. It turned out every even rural classroom had a DVD player, but the teachers would put it in and watch it. Of course, they were really not very skilled in the subject matter either; that was one of the fundamental problems.

So, rapidly we found that the teachers would get the DVD and they’d take it home first, and they’d watch it so that they understood it, and therefore when they came and the thing was presented to the class, they could at least help explain it or talk more cogently about it.

The next thing we found out was that they would take the DVD home and they would study it, not just to learn the material but to emulate the teaching manner of the great teachers. So, these teachers that had been largely written off as not very good actually were elevating their skills in teaching itself, as well as in the subject matter, and eventually many of them would come in and say, I don’t need the DVD anymore, I can give you as good a lecture as the person who was recorded.

So, I think that there are many, many ways that we explore using technology for some of these important things, and it always show that it’s very unpredictable about how this actually plays out, but it is very rewarding whether it’s in the field of health, education, or just rural productivity to see the important role that technology will play in many of these areas.

I don’t think that it’s possible to scale the current rich world models of health and education, for example, into 5 billion more people. We actually struggle in the United States to get health care for all Americans. The idea that we could somehow pay the bill for another 5 billion people when we can’t do our own 300-odd million people is clearly not going to happen.

So, it’s only through a mechanism that allows self-help to become the basis of bootstrapping this in these huge rural populations, and there’s only one scalable way to do that, and that’s to give them technology, just like DVD players and televisions. So, if we can really lower the cost of this through cell phones and low cost display technologies, I think we do have the ability to start that bootstrap going forward, and we’re very interested in that.

So, let me stop there, invite Ken Perlin to come up and join me on stage, and we’ll use the — Professor Perlin from NYU, of course, is responsible for a big project that we’re actually announcing here today, and I’ll let him talk to you a little bit about it before we do the Q&A. Thank you very much. (Applause.)

KEN PERLIN (Professor, Department of Computer science, NYU): Hi. So, yeah, it’s very, very exciting for all of us, myself and (Jan Floss ?), my co-director, and all the collaborators at the new Games for Learning Institute, to be working with Microsoft.

One thing I’ve said before, and it’s worth repeating is that one of the things that makes this institute so exciting is that Microsoft, when it did the original call for proposals, really asked us to do the science of what is effective in using games for learning. They didn’t ask us, oh, just make a game; they said, we really want to understand what will work so that this knowledge can be transferred to other games for learning, to other disciplines, to other age groups. It’s an extraordinary thing to be charged with, and we really have a chance to bring education into the 21st century.

So, I’d like to do something a little corny and let’s all thank Craig Mundie and Microsoft for this. (Applause.)