How autonomous systems use AI that learns from the world around it

If a mine collapses or an earthquake strands people underground in a subway car, first responders can’t rush into that unknown subterranean environment without potentially endangering themselves.

A rescue team must ensure an area is structurally sound and air is breathable before pushing forward — ­­which sometimes means help moves slower than anyone would like.

In a competition sponsored by DARPA, teams are designing autonomous robots that can explore and map these potentially dangerous underground landscapes and also identify objects of interest to first responders like survivors, backpacks, cell phones or fire extinguishers.

“With a robot, you’re able to take much more risk and potentially move much faster in a rescue,” said Sebastian Scherer, Carnegie Mellon University associate research professor and co-leader of Team Explorer, which took first place in the initial leg of that Subterranean Challenge using Microsoft’s AirSim technology to train its robots to recognize objects in a simulated mine.

“It’s really difficult to design a system to operate in an environment where you really have no idea what’s coming next. It has to be very robust and be able to make decisions on its own to get itself out of trouble,” Scherer said.

It’s exactly the kind of hard problem that Microsoft’s autonomous systems platform is designed to make easier — along with a growing list of other industrial and manufacturing applications in which AI can be used to teach machines to learn from and respond to the physical world.

At its Ignite conference in Orlando, Florida, this week, Microsoft announced that it is expanding a limited preview program of its autonomous systems platform, which will offer more opportunities for developers, engineers and enterprise customers to test its first components.

Autonomous systems are part of a new class of systems that go beyond basic automation. Instead of performing a specific task repeatedly without variation, they are capable of sensing and dynamically responding to changing environments to accomplish a desired goal. Microsoft’s platform uses a unique combination of machine teaching, reinforcement learning and simulation to help companies create these systems.

At Ignite this week, Microsoft also announced that it is launching new partnerships with companies such as MathWorks, a leading developer of mathematical computing software and maker of the MATLAB and Simulink products that are used by millions of engineers and scientists worldwide to design complex embedded and multidomain systems. The partnership will allow engineers to create autonomous systems using Microsoft AI and Azure with the widely used modeling and simulation tools already at their fingertips.

New partners also include Fresh Consulting, a Bellevue, Washington-based consulting team of designers, developers and engineers who are helping customers build new systems such as tractors that autonomously deliver materials to outdoor solar farms or construction sites.

Microsoft says these new partnerships reflect the company’s commitment to building an entire ecosystem of partners — with expertise ranging from simulation to drone software and systems integration — that will support customers interested in taking the leap from automated to autonomous systems.

In some cases, those pioneering companies are using autonomous systems to perform work that’s either too dangerous or tedious for people to want to do. In others, they are helping people who work with physical systems — someone tweaking chemical reactions to make an optimal batch of plastics or adjusting automotive components to ensure a smooth ride — make smarter decisions by processing greater volumes of information than people can comprehend.

Microsoft’s unique machine teaching approach allows employees who might know a lot about keeping large buildings at a comfortable temperature without wasting energy or how material needs to move around a construction site but who aren’t data science experts to more intuitively “teach” AI systems. By using their subject matter expertise to break complicated tasks into smaller parts, they help the AI hit on solutions faster. This approach also results in AI with explainable behavior, giving people a clearer understanding of how it makes decisions and confidence that the solution is sound.

Deep reinforcement learning algorithms — which the Microsoft autonomous systems platform selects and manages — learn by testing out a series of actions and seeing how close they get to a desired goal. But because no one wants to crash real robots or take critical pieces of equipment offline while the algorithms figure out what works, the training happens in simulated environments.

With the MathWorks partnership announced today, customers can now use MATLAB and Simulink models with Microsoft’s machine teaching tools for training autonomous systems on the cloud. This enables them to use a wider range of simulation models to instantly spin up scenarios that mimic all the different conditions that an autonomous system might encounter. Whether those represent the many potential hazards inside a collapsed mine or a jet engine flying in different weather conditions, this allows the AI to learn from many simulated situations at once.

The journey from automated to autonomous systems is a spectrum of solutions, and very few of the engagements we’re seeing are in that fully autonomous with no humans in the loop zone. The vast majority are assistive technologies that work with people.

Millions of engineers across industries such as automotive, aerospace, industrial machinery and medical devices have already built models of the systems they work on using MATLAB or Simulink. This new partnership allows users to bring simulation models built using MATLAB and Simulink to Microsoft’s Azure cloud computing platform, enabling unprecedented scalability and making it easier for developers and engineers building autonomous systems.

“Our core interest really comes down to engineering productivity — the ability to succeed at a task in the least amount of time possible,” said Loren Dean, MathWorks senior director of engineering for MATLAB products.  “This partnership allows engineers to stay in a familiar workflow to learn and apply AI without having to do the things that are non-traditional for them, like setting up the infrastructure to run a bunch of simulations at once. They’re shielded from all that.”

By running hundreds or thousands of simulations in parallel in Azure and learning from massive amounts of data at once, deep reinforcement learning algorithms can find optimal solutions to chaotic, real-world control problems that other types of AI still struggle to solve.

It turns out these problems are everywhere, said Gurdeep Pall, Microsoft’s corporate vice president for Business AI. Microsoft received three times more interest than it expected after opening its autonomous systems limited preview program in May.

The companies who have applied to work with Microsoft’s autonomous systems team and partners are looking to develop control systems to intelligently stitch fabric, optimize chemical engineering processes, manufacture durable consumer goods and even process food. The potential goes far beyond robotics or autonomous vehicles, Microsoft says.

“These are the kinds of diverse use cases for autonomous systems that we’re starting to see emerge,” Pall said.  “As customers learn about the capabilities of our toolchain, we’re seeing them apply it in really interesting ways because these control problems exist almost everywhere you look.”

Most customer use cases Microsoft has seen so far involve helping existing employees do their jobs more efficiently, safely or with higher quality, said Mark Hammond, Microsoft general manager for Business AI and the former CEO of the startup Bonsai, which Microsoft acquired last year. As sensors in modern workplaces collect ever more data, it can become difficult for any one operator — such as someone who is guiding a drill bit or calibrating expensive equipment — to track it all. AI tools can process that data and bring the most relevant patterns to that operator’s attention, enabling them to make more informed decisions.

“The journey from automated to autonomous systems is a spectrum of solutions, and very few of the engagements we’re seeing are in that fully autonomous with no humans in the loop zone,” Hammond said. “The vast majority are assistive technologies that work with people.”

Training AI systems in virtual worlds

Traditionally, AI models have often relied on labor-intensive labeled data for training, which works well for many problems but not for those that lack real-world data. Now, Microsoft and partners like MathWorks are expanding the use of AI into more areas such as those that require learning from the three-dimensional physical world around them — through the power of reinforcement learning and simulation.

Engineers have long used simulations to mathematically model the systems they work with in the real world. This allows them to estimate how a particular change in a chemical, manufacturing or industrial process may affect performance, without having to worry about slowing production or putting people or equipment at risk.

Now, those same simulations can be used to train reinforcement learning algorithms to find optimal solutions, Dean said.

“The AI is really augmenting how these traditional systems have worked — it just gives you greater confidence in your design and gives you additional capabilities that either had to be done manually before or were difficult to solve,” Dean said.

Imagine a building engineer whose job is to calibrate all the heating and cooling systems in a large commercial building to keep each room at a comfortable temperature as people stream in and out for meetings and outside weather fluctuates — while using as little energy as possible. That could involve tuning dozens of different parameters and might take many cycles of modeling and measuring changes for that engineer to find the best balance of controls.

With the new Microsoft and MathWorks partnership, that engineering expert could use machine teaching tools to help an AI system focus on the most important dimensions of the problem, set safety limits and figure out how to reward success as the algorithms learn. This allows for greater transparency and trust in how the AI system is making decisions and also helps it work more efficiently than randomly exploring all possibilities.

The engineer could train the AI using models that he or she already developed in MATLAB or Simulink. The simulations can be automatically scaled up in the Azure cloud — which means the engineer doesn’t have to worry about learning how to host and manage computing clusters.

The end result is the building engineer uses AI to zero in on promising solutions much faster — but still uses his or her judgment to decide what works best.

“This partnership really marries the best of MathWorks’ capabilities for modeling and simulation with the best of Microsoft’s capabilities for cloud computing and AI,” Microsoft’s Hammond said.

Our core interest really comes down to engineering productivity — the ability to succeed at a task in the least amount of time possible.

But simulation needs vary widely. Some engineers execute equations in pure math to model fluid dynamics while others want to test a drone’s detection capabilities in photorealistic scenes. That’s why the autonomous systems toolchain also includes AirSim, an open source technology developed by Microsoft to simulate vehicles, drones and other equipment operating in three-dimensional virtual environments.

In the DARPA Subterranean Challenge, for instance, researchers and students from CMU and Oregon State University used AirSim to train perception models to detect objects such as people, backpacks or phones from three-dimensional LIDAR data. In its two best runs, Team Explorer’s robots were able to detect and map 25 artifacts — more than twice as many as any other team.

Artists used reference material from real-world mines to create an intricate maze of virtual manmade tunnels in AirSim. The simulations also included the team’s robotic vehicles and sensor data that reflected obstacles and objects they might encounter.

Others have used AirSim to train drones to spot elephant poachers and autonomously inspect wind turbines.

“I can create one scenario with the wind turbine operating in cloudy skies with windy conditions and another in sunlight and hot temperatures and all the permutations in between,” said Microsoft Senior Principal Researcher Ashish Kapoor. “There are thousands of different worlds we can create instantly and in parallel that the AI can learn from.”

The ability to generate data on such a massive scale allows reinforcement learning algorithms to pinpoint exactly which actions or series of steps mattered most in reaching a goal, which is critical to solving dynamic real-world problems.

AirSim is also well-suited to modeling situations where people’s unpredictable behavior comes into play, Kapoor said. In those cases, there are no immutable laws of motion or physics on which to rely.

“We can’t use our existing control techniques, so you need something to create a world that mimics all this unstructured behavior. And that’s what AirSim does very well — it allows you to create data that represents the chaos of human life,” Kapoor said.

In the first leg of DARPA’s Subterranean Challenge, winning Team Explorer used Microsoft’s AirSim simulation technology to help build autonomous vehicles and drones that could map and locate objects in underground mines. Photo courtesy of Carnegie Mellon University.

Building an ecosystem of partners

Helping customers transition from automated to autonomous systems — a shift that Microsoft says is foundational to the fourth industrial revolution  — will require more than one company acting alone.

In addition to its strong portfolio of Internet of Things services, Microsoft’s partnerships with companies like Fresh Consulting will help customers who see the benefits that AI-powered autonomous systems could offer but who may need conceptual or technical help in building them.

Other collaboration and partnership announcements include simulation software makers AnyLogic, CGTech, solution providers Neal Analytics and enterprise drone software maker 3DR.

In a warehouse of today, said Fresh Consulting CEO Jeff Dance, companies have to build a lot of infrastructure around an automated robot arm that can only pick up an object of a certain size or in a certain orientation. They may create special shelving or pallets that need to be redesigned if the process or product changes. By contrast, autonomous systems are designed to adapt to the world around them.

“When you can create an autonomous machine that can deal with new situations it’s never seen before, you don’t have to create that infrastructure, and that’s a huge benefit,” Dance said.

Fresh Consulting is also developing a fleet of autonomous tractors that can deliver panels and other materials to workers assembling outdoor solar farms ­— as they’re needed in the field. With persistent labor shortages in the construction industry, that allows installations to go quicker and employees to focus on more interesting parts of the job, Dance said.

Microsoft is partnering with systems integrators like Fresh Consulting to help customers that are interested in autonomous systems but don’t have core expertise in hardware or software design, Pall said.

It’s still an emerging industry, and Microsoft is still in the process of listening to early customers to refine its autonomous systems toolchain and help put all the puzzle pieces together, he said.

“Eventually we will get to a platform that becomes absolutely accessible to everyone, and that will be the tipping point in how we build things the old way and how we build things in entirely new ways,” Pall said.

“But what we’ve seen through history is that it’s the folks who embrace these big leaps early, not the ones who move slowly until their comfort levels are hit, who emerge as leaders in the new paradigm. And you can definitely tell from talking to customers who is really leaping.”

Top image: Microsoft partner Fresh Consulting, based in Bellevue, Washington, helps customers build autonomous systems such as these tractors that can autonomously deliver materials to workers assembling solar farms. Photo courtesy of Fresh Consulting.

Related:

Jennifer Langston writes about Microsoft research and innovation. Follow her on Twitter.