Microsoft Challenges AI Hackers to Break LLM Email Service, Rewards Up to $10,000

Microsoft has launched an intriguing competition, inviting participants to test their cybersecurity skills by attempting to bypass defenses in a simulated email client powered by a Large Language Model (LLM).

Dubbed the “LLMail-Inject Challenge,” the event aims to evaluate the robustness of state-of-the-art prompt injection defenses.

With a prize pool of $10,000, the challenge is open to teams of up to five members.

Participants assume the role of attackers tasked with crafting emails capable of manipulating the LLM into executing unintended actions without user consent.

The LLMail service, designed for this competition, integrates an LLM assistant that helps users manage emails and perform tasks like sending messages.

However, it is equipped with advanced defenses against prompt injection attacks, making this a rigorous test of adaptive hacking techniques.

The competition is structured into 40 scenarios, each testing different configurations and defense mechanisms.

Understanding Prompt Injection Attacks

Prompt injection attacks exploit the instruction-following nature of LLMs.

Attackers embed malicious commands within inputs, tricking the model into executing unauthorized actions or leaking sensitive information.

In this challenge, participants must craft emails that bypass defenses while embedding hidden instructions for the LLM to follow.

The LLMail service employs multiple cutting-edge defenses such as “Spotlighting,” which marks data to differentiate it from instructions; “PromptShield,” a classifier that detects malicious prompts; and “LLM-as-a-judge,” which uses an LLM to evaluate potential attacks.

These measures aim to prevent attackers from successfully manipulating the system.

How to Participate

To join the challenge, participants must sign in using a GitHub account and form teams of up to five members.

Entries can be submitted via a dedicated website or programmatically through an API.

The competition scenarios vary in complexity, requiring attackers to adapt their strategies based on available information and system configurations.

This initiative is organized by experts from Microsoft, ISTA, and ETH Zurich.

While submissions are limited to this simulated environment, Microsoft encourages participants to apply their learnings in other cybersecurity contexts, such as its Zero Day Quest program.

By hosting this challenge, Microsoft underscores its commitment to advancing AI security.

The LLMail-Inject competition not only highlights vulnerabilities in LLM-based systems but also fosters innovation in developing robust defenses against emerging threats.

Also Read:

Microsoft Expands Windows Recall Feature to Copilot+ PCs for Insiders

See more