Microsoft says it needs to strengthen trust in AI, which cannot be achieved without a full understanding of the changes AI generates. In Microsoft’s AI Red Team, alongside cybersecurity experts, neuroscientists and linguists are analysing the ways in which fraudsters use AI to fool people.
The issue of cybersecurity is becoming increasingly relevant worldwide, including in Kazakhstan. In 2023, the country faced over 223 million attempted cyberattacks from foreign hackers. However, efforts to ensure data security are progressing: Kazakhstan ranked 78th out of 176 countries on the National Cybersecurity Index. The country’s index score was 48.05%, with the maximum possible being 100%, while the level of digital development of the population was 60.18%.
However, as AI advances, fraudsters’ schemes are becoming more sophisticated, requiring innovative approaches to prevent attacks. AI-based applications such as Copilot and ChatGPT are challenging our preconceptions about cybersecurity. What’s new in this area is that it’s no longer only necessary to have technological knowledge to counter fraudsters armed with AI; sometimes it’s also necessary to understand the human psyche and the social context.
Microsoft has recently set up a team called Red Team, specializing in curbing cybercriminal activity using artificial intelligence, with the idea that the defender must embrace a wide range of perspectives, including that of the criminal. Microsoft blogger Susanna Ray reports in detail on the composition and tasks of the team, which was set up in 2019.
The team, led by Ram Shankar Siva Kumar, brings together neuroscientists, linguists, national security experts, and a range of other professionals to assess societal damage beyond immediate security risks. This is necessary because generative artificial intelligence enables criminals and fraudsters to speak authentically, present narratives in multiple languages, manipulate undetected, or create highly lifelike images to deceive people or create social divisions and tensions.
“Cybersecurity, responsible use of AI, and the broader safe use of AI are different sides of the same coin,” says Siva Kumar. “A holistic, one-stop-shop approach is needed because it’s the only way to get a comprehensive view of the risks.”
Siva Kumar has teamed up with researchers from Microsoft’s Aether program (a research area on the ethical and societal aspects of AI). They are investigating how AI models, either through deliberate human intervention or by their own internal logic, can cause harm that has escaped the attention of developers and analysts. “Our activities cover a wide variety of threats, which we ourselves confirm the presence of,” explains Siva Kumar, adding, “We adapt quickly, we adapt our tactics frequently – that’s our recipe for success. We cannot allow ourselves to be forced into action by change; we have to anticipate it.”
The Red Team does not have a direct working relationship with the engineers developing the technology. Instead, they focus on those who are forcing AI-based systems to hallucinate to generate malicious, offensive, or biased content based on false or inaccurate data. Once the Microsoft AI Red Team finds a problem, it notifies the relevant AI Measurement team. This team assesses the threat that the problem poses to the entire ecosystem. Other internal experts then investigate the threat and look for solutions to the problem.Team members take on different personas, from creative teenagers interested only in pranks to criminals specializing in data theft and damage, to uncover blind spots in development and expose risks. Team members are drawn from all over the world, speaking 17 languages from Flemish to Mongolian to Telugu, to help uncover nuanced cultural contexts and identify region-specific threats. They not only seek to hack a system but also use large language models (LLMs) to launch automated attacks against other LLMs.
The group has also demonstrated its expertise by releasing open-source frameworks (e.g., Counterfit, Python Risk Identification Toolkit for generative AI, or PyRIT) earlier this year to help security professionals and machine learning engineers map risks. The team also shared best practices from their experiences with a professional audience.
“We’ve only been using generative AI for a short time, but we have already realized that we couldn’t do without the huge benefits it offers. However, it is now clear that a lack of confidence in AI tools could be a barrier to further progress. Microsoft has therefore mobilized enormous resources to ensure that the development and use of AI remains under control so that the positive societal impacts continue to dominate and the negative ones can be eliminated. Microsoft is ahead of the curve and ahead of its competitors in this endeavour,” said Renate Strazdina, Microsoft’s Regional Chief Technology Officer for the region, summing up the importance of the cybersecurity research work being done at Microsoft.
