Drones with brains and shelves with eyes — digitalizing the physical world

In the wind energy sector, the usual way to inspect wind turbines is to send a five-person team up each tower. The technicians use ropes and harnesses to make their way up and down the blades, which could be from 40 to 80 meters long. They visually inspect each one while dangling as high as 100 meters above the ground. Given that these are obviously windy conditions, it’s a dangerous activity prone to accidents.

Besides the danger to humans, the process is incredibly time-consuming. A minimum of six hours is required to fully inspect one wind turbine. Multiply that by the hundreds of turbines found in some wind farms, and you get a sense of the scale.

The team at Clobotics, a start-up dual headquartered in Shanghai and Seattle, wondered how they could do this better with new technologies. Clobotics is on a mission to digitalize the physical world using computer vision based artificial intelligence (AI) technology.

Drones were an option, but not just any drone. To manually pilot one from the ground to the turbine blades in these conditions would be too difficult. So, any drone would have to be autonomous, and its visual recognition capabilities would have to be incredibly precise.

So, Clobotics gave their drone a “brain.” Its autonomous control system can determine the position of the wind blades and perfectly track their whole length looking for defects. An onboard computer runs Clobotics’ computer vision, AI and data analytics software.

The vision system scrutinizes every side of the blade, looking for cracks as small as one by three millimetres. The resolution that has been achieved is so fine that the images even captured a tiny fly resting on one blade, its wings and legs clearly visible. The images are sent to the cloud to process, by using the Microsoft Azure platform. Clobotics’ data analytics and deep learning neural networks create a full report of any damage that needs to be addressed.

The system solves many problems. The first is safety. Using a drone means that humans are spared the risk of suspending themselves in hazardous conditions. The second is speed. Compared to five people spending six hours, one operator with a Clobotics drone can complete the process in 25 minutes, an efficiency increase of 10 times. And it provides valuable new data for wind farm operators. AI means the reporting is delivered eight times faster, and now there is a complete maintenance and inspection history of each blade of each turbine.

In this way, Clobotics is successfully achieving their mission of digitalizing the physical world, or “Di-Phy”.

On 17 October 2018, the company signed a strategic cooperation agreement with Shanghai Electric Group Co., Ltd., one of the largest power generation equipment manufacturers in China, and Luoyang Sunrui Wind Power Blade Co., Ltd., the leader in wind turbine manufacturing and maintenance service.

This official partnership represents a huge step forward for Clobotics as it takes on the international market. And it proves that traditional giants in the wind power industry are willing to adopt innovative technology like artificial intelligence.

Clobotics was founded in 2016 by four friends, all with extensive backgrounds at global tech firms. In their previous roles, the four worked with the passion to build the right platforms, making sure these were robust and valuable to the end users.

Now, as founder and CEO, George Yan says, “We’ve made ourselves become the ones to fill the gap between the platform and the customer needs by starting our own company. Microsoft’s cloud ecosystems are built by prolific technology partners who serve the end customers.”

Clobotics puts a premium on understanding customer needs and gaining specific industry expertise.

It’s a process they are also applying to the traditional field of retail. “The offline retail space is very much challenged by the likes of Amazon and JD.com etcetera,” says Yan, “We see the offline retailer guys need a lot of help.”

To offer this help, they trained their computer vision service on the consumer-packaged goods (CPG) industry. They developed a system that allows retailers to use smartphones to take photos of products on store shelves and get immediate feed-back. The AI-based technology identifies the SKU (stock-keeping unit) numbers at a rate of 95% accuracy, then gives real-time actionable insights like the shelf-share ratio, out of stock rate, and other meaningful data reported using Microsoft Power BI (Business intelligence).

This retail assistant is effective, but just the tip of the iceberg. Yan and his team have a “grand vision” to visually recognize every product in a store and use that to solve classic retail execution scenarios. The next step involves putting eyes on the edge, be it a refrigerated cooler or on store shelves.

Coca-Cola currently has more than a million fridges in stores around China. Imagine being able to use a new technology to transfer the physical layout, temperature, the number of times a door opens and closes, competitor product information, and more into digital information. This means Coca-Cola is adding more than a million point-of-sale business insights in real time.

Clobotics is making this possible by adding “eyes” to their fridges. Internet of things (IoT) devices take pictures of products each time a cooler door is opened and after it has closed. AI recognizes all the products inside and calculates what has been removed and added. It also detects if a cooler is being misused, like stocking competitor products. And all of this information is conveyed to the end user in real-time.

According to Yan, CPG companies estimate that the added visibility can help drive sales from 30% to 50%.

As these smart fridges roll out, Clobotics is working on the next phase — deploying IoT devices to watch every shelf in a store. Once their grand vision is achieved, and they can truly recognize and track every item in a store, the opportunities for retailers and CPG companies could prove remarkable.

It creates possibilities like a heat map, showing where products are flying off the shelves the fastest, or having the option to overlay demographic or pricing data across the store, all in real time. The system is sure to eliminate one of the most basic and pernicious problems of brick and mortar stores: empty shelves.

If you can find a way to digitalize aspects of the physical world, you can take advantage of the benefits that intelligent computer systems can offer. Clobotics is using vision to make it happen in wind and retail and CEO Yan hopes that “the technology we provide can be replicated across the world.”