LONG BEACH, Calif. — Feb. 11, 2010 — A Microsoft technical fellow and director of Live Labs, Gary Flake has made a career out of building technologies that glean insight from information.
Today at the TED2010 conference in Long Beach, Flake will show off Live Labs’ latest development — a data visualization technology called Pivot, designed to help people make better use of digital information.
“We’re at a really interesting point in the history of the Internet,” says Flake. “Pivot was not possible to build five years ago. But it is possible to build today.”
Pivot is an experimental technology that allows people to visualize data and then sort, organize and categorize it dynamically. The result is that correlations, exceptions and trends become immediately apparent in ways they can’t when information is stuck in rows and columns.
The program is designed to provide a much more natural way for humans to digest large palettes of information without losing their way — an idea that anyone who has analyzed giant spreadsheets may welcome. And while it has something in common with spreadsheets and many other technologies, Flake says Pivot is so new and different that it’s difficult to even picture what it does without seeing it in action.
“With Pivot you can interact with data in a way that is powerful, informative and fun,” Flake says.
Pivot combines related data — anything from pictures, videos and maps to batting averages and financials — into large collections that can then be manipulated, sorted, filtered and examined visually. In this way, the data itself can help shape and inform the way it is presented.
Perhaps Pivot’s most compelling feature is its ability to smoothly and quickly arrange collections according to common characteristics and then zoom in for a closer look, by either clicking on a particular item or filtering the collection to get a subset of information.
“With Pivot you can swim through the data, taking little twists and turns,” Flake says. “If you are looking at all the information at once, the proverbial forest, you can click on any one item or filter and smoothly zoom into the trees without any interruption.”
The example above of a Pivot view makes it clear how pitchers outnumber other players on Major League Baseball teams. Subsequent views can display salary, performance against salary, and other information on the fly.
Flake says that Live Labs’ research with users has found that the continuity and smoothness Pivot provides in surfing through data is important to help users understand what they’re looking at, and how they got there.
“We found that if you make it a sudden transition, people lose their way,” he says. “But if you make it very smooth and continuous, people have a mental model of how they got to where they are.”
Because Pivot works with almost any kind of data, its potential uses are as varied as the types of information available today — in other words, practically infinite. A Pivot collection designed to help study the history of movies, for example, could sort by male or female leads, and then sort again to find their most frequently occurring co-star. The user could then “pivot” the information again, to shape the display of the movies themselves, perhaps by decade.
Realizing that the horizon for Pivot’s possibilities was far beyond what one research group could hope to imagine, Live Labs released the technology on a limited basis last fall at the 2009 Professional Developers Conference. After only a couple of months, says Flake, the examples started rolling in.
“Just as we anticipated,” Flake says, “people are using it in ways that we never anticipated.”
One of the first projects Flake saw was a Pivot collection that a user had set up against the information on his Facebook page, displaying all the relationships and correlations among his group of friends — Who lives where? Who is married? What is the age distribution?
Another collection drew on pictures and vital statistics across a dating Web site — allowing users to sort by age, for example, and view all profile photos against every age group.
More serious uses of Pivot included a school district study that showed investments against test scores, attendance rates, teacher salaries and other data related geographically to local schools. The collection mapped out correlations between high-performing schools and high-performing students — with the hope of lending insight into difficult policy decisions.
Yet another early example came from Microsoft’s own internal marketing group, which used Pivot to examine its Internet advertising over the past year.
As director of reporting and digital analytics within Microsoft’s global marketing organization, Michael Moore monitors click-through rates and other measurements to determine the effectiveness of ads placed on the Internet. Moore says a real strength of Pivot is its ability to combine hard data, such as click-throughs, with images of the ads themselves, a feature he says facilitates a subjective analysis of the underlying objective data.
“Pivot is a beautiful merge of quantitative and qualitative analysis,” Moore says. “You can associate copy, layout and images, the qualitative piece, with metrics data from the business intelligence system.”
The marketing group’s use of Pivot led to some surprising insights, Moore says, such as the fact that images of faces were more effective in larger ads than in smaller ones, or that ads containing messages around the concept of “simplification” tended to pull in more traffic than others.
Moore also cites the ease with which his team was able to create a fully functioning Pivot collection. Working with a developer familiar with XAML, the group was able to tap Pivot directly into its content management system and automatically build the collection of ads being served to the Internet. The result is a living, breathing business intelligence system that only took about a month to design and build.
What’s more, when it came time to present the results to senior management, Moore’s group used Pivot in yet another way that the Live Labs team hadn’t anticipated: as a dynamic presentation tool.
Says Flake: “Imagine if instead of putting up a spreadsheet or a pie chart in a meeting with your boss, you can present a living infographic. You see one view of the data, then take a little action and the data pivots to a different view.”
Live Labs Director Gary Flake will present Pivot at the TED2010 conference today in Long Beach, Calif.
Flake says Moore’s use of Pivot solves a problem that literally every business faces: How do you tease out insights to inform decisions that help you become better at doing business? And that concept, he says, makes Pivot potentially relevant to almost any organization that wants to improve what it does.
“We presently have more data than we know what to do with,” he says. “It’s occurring in the biomedical domains, it’s occurring in government initiatives. The challenge going forward is, how do we use that data to educate and inform ourselves, so we can make better choices?”
It’s a question that’s too large for any one lab to examine. So to help tap into the collective creativity of its audience, Live Labs has made Pivot downloadable to people who want to give it a try and think about new scenarios it enables.
Meanwhile, as he prepares to present Pivot to an audience of scholars and technology insiders at the TED Conference this week, Flake emphasizes that Pivot is still an experiment, still a research project in its own right. But he finds himself wondering what lies around the corner as Pivot gets into the hands of more people.
“Imagine if you can take this data and actually start connecting it with things like disease symptoms or complicating factors like lifestyle choices that people make,” he says. “With a tool like Pivot we’re hoping that people, for the very first time, will be able to work through large volumes of data in a way that makes insights pop — where suddenly we tease out some pearl of wisdom from what was previously mysterious or obfuscated, and that insight allows us to make a better decision or create a better policy.”