You could say it all started because Sam Sun does not like shopping.
Sun’s team at Microsoft had been working on a feature for the Microsoft Edge browser that would improve the online shopping experience through web entity extraction. It could pull relevant data from various sites, allowing the user to compare prices, find deals and see when something goes on sale or comes back in stock, for instance.
But Sun didn’t see how such a feature would be all that helpful for him personally, and he and his team started to question whether there was a way to use the architecture to create a more versatile tool that could help more people with a wider array of tasks.
That question was answered by Sun’s team during this year’s global Hackathon, an annual event produced by the Microsoft Garage. The 13-person team, led by Sun’s day-job colleague Steven McMurray, created a customizable digital assistant – built as an extension for Microsoft Edge – that a user can teach to do any number of tasks. That project was just awarded the Hackathon grand prize.
So what exactly could a customizable assistant do?
McMurray said he would train it to keep him posted on the pollen count of ragweed, an allergen to which he is particularly sensitive.
Sun would use it to find and sort research.
Team member Kelsey Juraschka would use it to keep tabs on national park reservations and nab a highly sought-after permit.
Other team members said they could book a tee time as soon as one becomes available, aggregate the count of unread messages across an array of social media platforms or track the stock market.
Their varied ideas for applying the tool demonstrate just how useful a digital assistant could be if the user gets to show it exactly what to do.
So how would you use it? The team is offering a chance to share feedback and request an invite for early access to the project (available in the U.S. only).
‘A window to the world’
Because several members of the Hackathon team work on Microsoft Edge in their day jobs, they were uniquely positioned to understand the diverse needs people have when it comes to using a browser, according to team member Guru Singh.
Singh said he views the browser as a “window to the world.”
“You have access to all kinds of resources,” he said. “Some of the things we do every day are monotonous and could be automated. We thought we could transfer the power to automate to the people with relatively little effort.”
McMurray said it was Sun who pointed out that it was unlikely the Microsoft Edge team would be able to figure out all the different things people may want to do online, and Singh who mentioned that the team would have to focus on the concept of teaching if they really wanted to help people.
“People are heterogenous,” Sun said. “Not everyone wants their personal assistant to perform the same small set of tasks.”
We thought we could transfer the power to automate to the people with relatively little effort.
The set of tasks that digital assistants and smart devices know how to do for us is small because each one must be specifically programmed. That means only those skills that are deemed useful to a broad range of users will ever make it to a product.
To bypass the engineering bottleneck, this assistant starts with no preset knowledge.
“It’s completely a blank slate that you work with and teach it how to help you,” McMurray said.
That teaching is done by each individual user through machine learning. Unlike traditional machine learning, which relies on large quantities of data for accuracy, each user’s model can learn quickly with very little information because all the data the user provides is accurate and highly relevant to that particular model.
The approach puts the consumer in control.
“You don’t have to be a coding wiz,” Singh said. “We don’t think that automation should require a sophisticated knowledge of how this stuff works.”
Help me help you
Let’s say, for example, you want to keep track of that pollen level. You would navigate to the website where that information exists, click on the specific data you want to track, then save the page as a pin within the assistant extension.
The pollen level will then display on your personal assistant dashboard, which will update as new information is available. It might be one of a dozen types of information your personal assistant tracks and displays, from package tracking to unread emails to campsite availability. And you can ask to be notified if that pollen level hits a certain value or have your assistant automatically book you a campsite and let you know.
Those notifications could come on the browser itself or via a not-yet-built mobile app, made possible because the browser is running in the cloud and uses WebDriver to simulate basic browser navigation such as clicking and scrolling. For tasks like checking email, the user’s cookies would be securely passed to the cloud and used to refresh the data.
But, of course, your new assistant is just getting to know you and your needs, so it may not get everything right on the first try.
That’s why another critical piece of the project is a user’s ability to search for information using their own natural language, and to continuously provide feedback to train the personal assistant to become more useful over time.
So, if you type “What is the ragweed pollen level today?” into the search bar on your personal assistant dashboard – or, eventually, ask it out loud through your app – your personal assistant would display what it thinks is the information you’re after. If it gets it wrong, you tell it.
Over time, the personal assistant not only gets more accurate, but learns your specific style of searching.
“The trainability of this model is really compelling,” said Jeff Ramos, who heads the Microsoft Garage. “It could really set Edge apart in the browser space and even automate tasks across platforms.”
An ideal incubator
The Hackathon gave the team the opportunity to try something that they weren’t completely sure would work. That’s not a new experience for McMurray, who worked at a startup for three years before joining Microsoft. He brought that energy to his role at Microsoft, and particularly to this project.
“We had the ability to think freely and openly and to take big risks and not be afraid to fail,” McMurray said. “Let’s just build something cool and see if it sticks.”
That freedom is a big reason the annual event continues to draw so many people. This year, the Hackathon was all virtual, and had more than 66,500 participants. To encourage increased participation, the Garage added features to its internal website HackBox, added a new online resource portal and video stream, and gave people the opportunity to contribute in new ways beyond traditional hacking: They could volunteer to be consulted as subject matter experts, offer project ideas or weigh in on existing projects.
Ramos said those changes were a boon for inclusivity.
“Our digital presence made people feel like Hackathon was more of an even playing field for folks who were working outside of large development centers,” he said.
We had the ability to think freely and openly and to take big risks and not be afraid to fail.
The all-virtual format did present some challenges, even for the Redmond-based winning team. Sun, who has done Hackathon almost every year, said brainstorming is easier face to face. Team member Nancy Li, who participated in two previous Hackathons as a Microsoft intern, said the team had to be much more proactive about checking in with one another. They both credit McMurray with creating clear work groups and tasks that were delineated from the outset.
All the team members are quick to acknowledge one another’s contributions, and it’s clear that they had the trust and confidence necessary to work in parallel on their individual project components before stitching it all together.
“The Hackathon is a great outlet for people to take the skillset they have deeply in one domain and apply it to a passion interest they have,” said Ed Essey, director of incubation for the Microsoft Garage.
And, in some cases, the Hackathon is an opportunity to learn something completely new. Li, who doesn’t work on the Microsoft Edge team in her day job, learned Javascript so she could help with debugging. She also played a big role in putting together the team’s demo video.
Singh wasn’t sure what to expect from the Hackathon, his first at Microsoft. He was pleased with how collaborative the experience was.
“All these people from different backgrounds were coming up with ideas,” he said. “It brings out the diversity of thought and diversity of approach.”
The project is an acknowledgment of consumer individuality, said Essey, who will now work with the team closely to explore what happens next for their idea.
“You can aggregate these things that take all of our own needs into account while tying into all these amazing sources around the world,” he said. “The time is right for that kind of integration.”
Sign up for project updates here.