UNSW explores AI infused analytics for dual data lakes

UNSW has begun exploring how AI might be able to help it extract additional value from its dual data lakes. The University built its first data lake in Microsoft Azure in mid-2019 and has since used Azure Databricks to create a second cloud-based curated data lake to accelerate time to value.

Business functions across UNSW have been able to access the curated data and build their own reports using Power BI. This democratised access to a single source of truth has been essential to help the University respond effectively to the challenges posed by the COVID-19 pandemic.

It is now exploring how artificial intelligence could be used to glean insights even faster, and how it might be able to use voice commands to access information.

The program of work is led by Kate Carruthers, chief data and insights officer, UNSW, and one of the recently named Global Top 100 Data Visionaries for 2020.

According to Carruthers; “I want us to be able to talk to our data. I want us to be able to, like in Star Trek, ask ‘how many students do I have’ and have the computer understand that and tell us the answer. What we’re building is basically the building blocks to get to that point.

“Voice recognition is coming along so fast that I think that that’s going to be a huge area for innovation.”

UNSW has also turned to AI to help manage the issues related to the need to model the student load in these times of Covid19. The team is under pressure to build complex models with extremely tight turnaround times. With the impact of Covid19 on international students coming to Australia this is a hot issue for the University. UNSW is working with a Microsoft partner, Insight, to develop new ways to model the data in the Azure cloud and to use Data Bricks and the new Microsoft Project Bonsai AI toolset to deliver new what-if analysis models driven by ML.

Additionally, Carruthers is exploring how Machine Learning (ML) DevOps could be leveraged to develop solutions to provide learning analytics that reveal how students learn best and also identify any blockers.

“We’re onboarding the Moodle data for this proof of concept in the data lake. That will then enable us to then use ML techniques to then do some learning analytics,” said Carruthers.

Tiffany Wright, Director Education, Microsoft Australia, said the work being undertaken by UNSW was ground breaking for the sector and comes at a time when all Universities are facing unprecedented challenges and an urgent need to transform the way they operate.

Wright noted also that Carruthers has recently been named a Microsoft Regional Director for cyber security, which will see her provide coal face insight to help steer Microsoft thinking with regard to cyber security strategy and product development.

“The work that Kate is leading at UNSW with AI and machine learning has the potential to be a game changer in terms of delivering instant insights to people across the university – whether they are a lecturer, a researcher or an administrator.

“UNSW is fully cognisant of the critical importance of data integrity and security to that work.  The University’s data collection includes everything from personal student data to research data, satellite data to climate data.

“All the data stored in UNSW’s two Azure based data lakes is encrypted, and the data is also encrypted when being transferred to and from the data lake,” said Wright. “This helps to ensure the integrity and security of the data, and hence the value of analytics performed using it.”