AEMO sparks data driven, AI infused energy transformation

Microsoft is delighted to have partnered with the Australian Energy Market Operator (AEMO) on its nationally important digital transformation designed to drive efficiency and consumer value in a transitioning energy market.

An Azure-based big data solution is now being market tested that will next year allow Australia’s energy market to transition from 30-minute to five-minute settlements, encouraging increased competition and driving innovation.

Further, AEMO is also exploring the use of artificial intelligence and machine learning to enhance its analytics capability and build new classes of digital tools. This includes a next-generation simulator that can provide near real-time support for system security challenges in a rapidly evolving power system

Established in 2009, AEMO is Australia’s independent energy markets and power systems operator, and system planner.

One of its principal responsibilities is settlement of the $16 billion-plus National Energy Market (NEM) which connects the grids of eastern and southern Australia states and territories to create a wholesale energy market. Retailers and wholesale consumers pay AEMO for the electricity they use, and AEMO then pays the generators.

In the past AEMO has only been able to settle the market in 30-minute blocks because that was the limit of its ability to segment its data; the solution developed with Microsoft in association with Tata Consultancy Services uses an Azure cloud based solution to allow settlement in five-minute blocks, which should provide a stronger incentive for market participants to respond to the rapidly changing dynamics of the electricity market.

Currently, the power to the National Energy Market is dispatched in five-minute lots – but only settled every 30 minutes.  The switch to five-minute settlement (5MS), scheduled for October 2021, aligns the dispatch and settlement times and is expected to remove a number of potential market barriers for renewable energy providers for bidding and dispatching as well as encourage additional innovation.

Steven Worrall, managing director Microsoft Australia said; “Energy reliability and marketplace competition are two critical inputs to a strong national economy. It has been a privilege to work with AEMO on this nationally important digital transformation that lays the foundations for ongoing innovation.

“The move to fine minute settlement also sets the stage for even greater innovation in renewables. Energy sustainability is an issue close to Microsoft’s heart as we have been carbon neutral across the world since 2012, and have committed to being carbon negative by 2030. The transformational work that agencies like AEMO are undertaking takes all companies an important step closer to operating in a truly sustainable way.”

AEMO has been working with Microsoft and partne Tata Consultancy Services is to transition from a legacy settlement system to its big data Metering Data Management solution built on Microsoft Azure cloud. Cosmos DB is the main data store, leveraging Azure Kubernetes Services for the application and runtime layers.

There are more than 10 million meters connected in the retail and transmission network. With meter data storage requirements being three years, and the shift to 5MS, the volume of data to be stored will soar from 90 billion reads to around 2 trillion by 2025.

AEMO’s digital transformation also includes leveraging Power BI to accelerate the production of reporting, leading to improved internal operational efficiency. The sheer volume of data now available completely transforms the market reporting that will be possible. Databricks is being used for the business intelligence and reporting services.

“AEMO’s increased operational efficiency in the management of data and reporting is already yielding efficiencies with report build times, with no IT needed,” said Locandro.

The organisation is now gearing up to leverage artificial intelligence and machine learning within AEMO and build new classes of digital tools to enhance forecasting and grid reliability.

As an example, AEMO is working on an energy simulator for the entire Australian east coast network, incorporating detailed models of wind and solar farms, and battery storage systems. It simulates their behaviour when connected to the grid to ensure the overall system is not negatively impacted, and the grid’s operation is simultaneously optimised for security and efficiency.

According to Locandro; “At the new required level of modelling detail, running a one-minute simulation currently takes upwards of two hours of processing time – it is highly compute intensive. We’re going to develop a simulator that will take it down to about three minutes to run a one minute of simulation without loss of fidelity. What that means is during a bushfire whereby powerlines are switched off or damaged, we will be able to quickly simulate and do ‘what-ifs’ to plan mitigation strategies.

“The simulator will be built in a hybrid platform, using cloud-based access to allow industry, researchers and developers to simulate the grid in an extremely fast fashion, which has never been done before.”