From gaming system to medical breakthrough: How Microsoft and Novartis created Assess MS

When Microsoft released the Kinect system for playing Xbox video games about five years ago, it attracted the interest of an unlikely source: the healthcare company Novartis.

For years, Novartis has been trying to find more consistent ways to quantify whether the treatments it is developing for multiple sclerosis are working, but assessing whether a patient’s symptoms are stabilizing or getting worse is complicated.

That’s partly because multiple sclerosis itself is complicated: In some patients, symptoms might progress with heartbreaking speed, while in others they may show up slowly, erratically and over a period of many years.

“One of the most difficult things about MS is the uncertainty of it,” said Cecily Morrison, a researcher in Microsoft’s Cambridge, UK, research lab, who has spent the last couple of years working on the research project that was borne out of Novartis’s idea, dubbed Assess MS.

To try to quantify the progress of multiple sclerosis, doctors have developed a standard set of tests they perform, like asking a patient to touch their nose or sit with their arms outstretched. Doctors watch the patient and then use a rating scale to determine how strong the patient’s symptoms are.

The problem? Doctors are only human, and despite all their best efforts to standardize the MS test, in the end it is subjective. The researchers found that when a group of doctors are shown the same patient doing the same movement, some may interpret it as a “1” on the rating scale, while others will say it’s a “2.” Even when the same doctor is shown the same movement on two different days, that doctor may give that patient a different rating.

“The clinicians that we worked with really care about their patients. They really want what’s best for them, and even the best neurologist will admit that when they use these rating scales, it’s pretty coarse-grained,” said Abigail Sellen, a principal researcher in the Human Experience and Design group at Microsoft’s Cambridge, UK, lab. “They know that there’s a lot of variability, even in their own judgments, over time.”

That’s why the possibility of using computer vision, which is the type of technology found in the Kinect system, was so intriguing. Using a tool like the Kinect, the researchers at Novartis figured they could get a more consistent reading of how a patient performed on each of the tests, bringing a new level of uniformity that would help doctors better assess the progress of the disease. That, in turn, could speed up the process of getting the right treatments to patients.

The goal was not to replace the doctor but rather to augment the doctors’ knowledge of the disease with a more consistent measurement of the symptoms, in the same way that an ophthalmologist can measure a patient’s declining eyesight.

“What you don’t want to do with these systems is replace the expert. You want to bolster the expert,” Sellen said. “What we’re doing is giving them a set of data that they can then weave into their judgment.”

Machine learning, and learning to work with machines

The computer scientists who work within Microsoft’s Technology and Research organization were immediately excited by Novartis’s proposal.

Microsoft researchers have long been at the cutting edge of a field called machine learning, which is a branch of artificial intelligence in which systems get better at doing something as they gather more data. Machine learning is ideal for a project like Assess MS because, as the computer vision system captures more recordings of patient movements, it can deliver more consistent results showing the disease’s progression.

Novartis’s challenge offered an opportunity to break new ground in the field of machine learning while also helping the more than two million people who have multiple sclerosis get treatment faster. But in order to do so, they would have to come up with some entirely new ways of training systems and getting results.

“It was clear that this was a very ambitious project,” said Peter Kontschieder, a postdoctoral researcher at Microsoft who has built many of the machine learning algorithms used in the Assess MS project.

In a typical machine learning scenario, an experiment starts with a huge amount of data, such as a bunch of pictures of trees. The computer is given those pictures and, using a machine learning algorithm, it creates a model that recognizes what a tree looks like. Then, the next time it sees a picture of a tree –even if it wasn’t one of the pictures it was originally shown – it has learned to identify the image as a tree.

For the Assess MS project, the first challenge was gathering data.

In order to get an accurate view of how an MS patient was faring, they needed to figure out a way to make the Kinect depth camera, which is designed to recognize the sweeping gestures of playing a living room video game, pick up on subtle movements such as the sway of a patient’s torso.

That required the researchers to come up with entirely new algorithms that would work behind the scenes with the Kinect’s depth camera, and that would both recognize the patients’ body parts in a more nuanced way and provide a more precise representation of how the patient was doing on each of the tests.

Even once they had created those algorithms from scratch, the machine learning experts still knew they wouldn’t be able to collect as much data as they would ideally want to have to create a machine learning model. That’s because they were gathering these types of recordings for the first time, and they would be limited in how they could use the data by stringent patient privacy protections.

There was a third challenge as well. Unlike simple image recognition, in which you can say one picture is a tree and another is a car, this project required the researchers to grapple with nuanced, subjective data showing even slight disease progression.

All of these challenges required new strategies for labeling the data consistently, so that they could create a really strong model for quantifying the results accurately.

“We had to think hard about better, more accurate algorithms that would really pick up on the subtle signals that differentiate between an MS patient and a healthy individual,” said Antonio Criminisi, a principal researcher in machine intelligence and perception at Microsoft’s UK lab.

A collaboration between doctor, patient and technology

As Kontschieder, Criminisi and the other machine learning experts were working on these brand-new algorithms, another group of Microsoft researchers was tackling a different element of the project.

In order for Assess MS to work, they needed to create a system that would be easy for doctors and patients to use in the real world.  That meant the Kinect-based system they were creating had to fit into an exam room, which might be small or crowded or have things like chairs in the way, and it had to be easy for doctors and nurses to operate.

“We needed to ask, right from the beginning, ‘How do we make it work in the clinics?’” said Morrison, a computer scientist whose expertise is in human-computer interaction.

That seems logical, but Morrison said one common pitfall is that researchers oftentimes design tools in a laboratory setting, with none of the constraints of a real doctor’s office, and then wonder why the technology doesn’t work in practice. That’s why the researchers began working early on with clinical partners atUniversity Hospital Basel, University Hospital Bern and VU University Medical Centre Amsterdam.

“It needs to be a collaboration between the patient, the clinician and the technology,” Morrison said. “What we try to do is strike a balance.”

With Assess MS, for example, the human-computer interaction team found that some medical professionals stood in front of the patient to demonstrate the movement, and in doing so got in the way of the camera. So they designed a secondary screen that could move behind the camera, subtly guiding the clinicians out of the way while still allowing them to monitor the patient.

They also created a simple animation of a human that would demonstrate the movement, easily guiding the patients to do the movements and also ensuring they were well-positioned for the camera to pick up the movement accurately.

Finally, they added a way for the clinician to see if something was in the way of the camera, such as a handbag on the floor nearby. And they added wheels to the machine itself, so it was easy to move around if something was in the way.

“We designed the prototype to help the clinicians help the computer, so that they could get the data they want,” Morrison said.

A lot of work, and a lot of possibilities

After years of working on design and algorithms, the researchers say they have developed a proof of concept, using a limited number of patients, so they know the system works in principal. The next step is to test Assess MS in practice, so they can see how it works with a broader number of patients.

In the meantime, Kontschieder and his colleagues will continue to improve the algorithms they use for measuring patients’ symptoms, because they’ll continue to get new data. In that sense, Kontschieder notes, the work on this project is never truly done.

Ultimately, the researchers hope that Novartis and other pharmaceutical companies can use Assess MS to speed up clinical trials for multiple sclerosis, and perhaps, eventually, for other, similar diseases as well.

“Novartis is leveraging digital technologies to transform patient care and drug development,” said Vas Narasimhan, global head of development at Novartis Pharmaceuticals. “We are excited about our collaboration with Microsoft Research to develop Assess MS, a more consistent way to measure motor dysfunctions caused by multiple sclerosis, which could lead to the development of better therapies and care for patients.”

Related:

Microsoft-led research team wins Marr prize for outstanding computer vision research

Molecular biology meets computer science in new system for CRISPR

The long quest for technology that understands speech as well as a human

Happy? Sad? Angry? This tool recognizes emotion in pictures

Lead image: Microsoft researcher Cecily Morrison demonstrates Assess MS, a tool that aims to bring more consistency to tracking the progress of multiple sclerosis symptoms. (Photo by Jonathan Banks)

Allison Linn is a senior writer at Microsoft Research. Follow her on Twitter.