Fetch! New Microsoft Garage app uses artificial intelligence to name that breed

Man’s best friend has inspired a new app – Fetch! Using your iPhone camera or photo library, it can identify and classify dogs by breeds and tell you what kind of human personality fits best with specific breeds. And just for fun, the app will even take an informed guess on what kind of dog you or your friends might be.

Released through the Microsoft Garage just in time for the American Kennel Club’s Meet & Compete and the Westminster Kennel Club Dog Show, this mobile app demonstrates the potential for Microsoft  researchers’ continued advances in artificial intelligence, which have already appeared in other playful ways through Microsoft Project Oxford-powered experiences such as HowOld.net, TwinsOrNot.net,  MyMoustache.net and Mimicker Alarm. In Fetch!, Project Oxford works together with some powerful new machine learning technology to deliver interesting results for all kinds of photos.

“There was an interest in creating a framework that would allow you to take a domain – in our case, dogs – and recognize numerous classes, such as breeds. We were interested in enabling an app to allow you to make object recognition extraordinary, fun and surprising,” says Mitch Goldberg, a development director at Microsoft Research whose Cambridge, U.K based team built the experience. His team works at the intersection of user experience, machine learning, computer vision and more recently, intelligent cloud services. He’s also had two German shepherd dogs, though now he has a cat. “We wanted to bring artificial intelligence to the canine world. We wanted to show that object recognition is something anyone could understand and interact with.”

Fetch! is designed for repeat use, and after giving it a couple tries, it’s easy to see how addictive it can be. You start with your dog, or your friends’ dogs. If the dog’s breed is unknown, the app will show a percentage of the closest breed. Tapping the percentage rosette leads to the top five breeds that could be in the dog. Clicking on the arrow in the corner leads you to more information on the breed.

“If you want to take photos of dogs, it will tell you what dog breed it is, if it’s one of our supported breeds,” Goldberg says. “If I choose to take a photograph of a flower, it’ll say, ‘No dogs found! Hmmm… This looks more like…flower?’ But if you take a picture of a person, it’ll kick into its hidden fun mode. And in a playful way, it’ll communicate to you not only what type of dog it thinks you are, but also why. It’s fun to see if the app knows it’s not a dog. A lot of the time, it’ll tell you what that image is. When there’s not a dog, you still want to use it.”

No two pictures yield the same result. You could resemble a Doberman Pinscher in one photo (sunglasses, no makeup) or a Pekingese (no glasses, makeup) in another. If you photograph an inanimate object, it might tell you, “No dogs found!” and make an informed guess at what it is.

If you like what you see, you can share the image on your social networks and through email.

Two years ago, in July 2014, a demonstration of the technology behind Fetch! appeared on stage at the 15^th annual Microsoft Research Faculty Summit. Dogs stole the show, as they always do.

Even then, the technology was able to tell the difference between people and dogs, with Cortana saying, “I believe this is not a dog.”

In the two years since that demo, a lot has happened.

The team has developed a website, What-Dog.net that has photos you can play with to find out about the breeds, and you can also submit your own photos and share them.

But the app is really where the most progress is apparent. Simply but stylishly designed, it’s incorporated a lot of user feedback from dog owners within the company.  It’s full of features, including a scrapbook where you can keep track of all your pictures and results. There’s also an impressive list of breeds that contains information such as disposition, size, coat and what types of families are best suited for each. The team consults several dog experts and kennel clubs to curate and update the information about dog breeds, not relying solely on gathered datasets.

“What makes this unique is the combination of machine intelligence and expert data about dog breeds, which helped us be more accurate. We invested in dog experts in making sure we had correct images for each breed,” says Javier Alvarez-Valle, senior software development engineer on the team that built Fetch!.

For Alvarez-Valle, the only member of the team who hasn’t had a dog, the app tuned him into looking at dogs while he was out and about, and trying to guess what breed they were.

“This is the kind of app you’re going to take out when you’re with your friends,” he says. “It’s a lot of fun to learn about what dog you look like, and it comes up with surprising comparisons. It will also make for good discussions when I post on Facebook. I think people will find many more uses than we can anticipate now.”

Alvarez-Valle says that one of the biggest challenges was turning a research project into a product for the real world.

“It’s hard work to classify any number of breeds,” Goldberg says. “One of the things about using deep neural networks is that you don’t train it on a particular dog breed. You give it numerous images, and it, in the training process, determines what’s unique in each of the dogs. We use one of the hottest machine learning techniques available, called deep neural networks. In fact, there is very advanced work underway at Microsoft in this area, which are able to take apart subtle differences, even when breeds look similar or through the many different colors within breeds. Every time we add more, that’s the beauty of the deep neural network in understanding new, unique breeds. This is a really complex problem.”

Goldberg also credits the Garage with making the project a reality.

“The Garage provides a really great way to share our work with the world quickly,” Goldberg says, “and to discuss our project with passionate users, obtain some valuable feedback, and serve as a channel to get our project into the hands of people who love animals.”

It’s early days for this technology, the team acknowledges, and they’re encouraging user feedback to constantly improve the app, which is powered by machine learning and capable of continually getting better the more it’s used. “A feedback feature is built into the app. So if you’re sure your dog is one thing when the app says it’s another, send us that info. Every image improves it.” says Chris O’Prey, another software development engineer with the Fetch! team.

“We’ll be able to look at trends and see what breeds we’re classifying incorrectly,” adds Goldberg. “And we’ll be able to iterate and improve our classifier.”

“I see Fetch! as a great example of how a carefully crafted user experience can enhance the power of machine learning,” says Ken Woodberry, deputy managing director at Microsoft’s research labs in Cambridge.

Download Fetch! for free from the Apple App Store or via What-Dog.net and learn more about it at the Microsoft Garage. You can find more information about Project Oxford AI services here.