Researchers Unveil 10 Prompt Hacks That Bypass GenAI Guardrails

As generative AI adoption surges across industries, a new research report from Pangea, a leading AI security firm, has spotlighted the growing threat of prompt injection attacks and the urgent need for robust defenses.

The findings, emerging from Pangea’s global $10,000 Prompt Injection Challenge, offer the most comprehensive empirical analysis to date of real-world attack techniques and their impact on enterprise AI applications.

Throughout March 2025, more than 800 participants from 85 countries submitted nearly 330,000 prompt injection attempts, utilizing over 300 million tokens.

The challenge tasked AI hackers with overcoming a series of increasingly sophisticated security guardrails in three virtual rooms, generating a dataset that exposes critical blind spots in current AI security practices.

Key Findings: Unpredictable and Evolving Threats

• Non-Deterministic Security: Prompt injection attacks display unpredictable success rates. Even when a specific attack fails repeatedly, it may suddenly succeed due to the inherent randomness in large language models (LLMs).
• Data Leakage and Reconnaissance: Beyond leaking sensitive data, attackers can use prompt injection for adversarial reconnaissance, uncovering details about server environments and accessible resources.
• Vulnerability of Basic Guardrails: Approximately 1 in 10 prompt injection attempts succeeded against basic system prompt defenses. Multi-layered protections, however, reduced successful attacks dramatically.
• Agentic AI Risks: As organizations integrate LLMs with databases and external tools, compromised systems could enable attackers to move laterally within networks, amplifying the potential damage.

The challenge’s only full victor, penetration tester Joey Melo, spent two days crafting a multilayered attack to bypass all security layers, underscoring the sophistication and persistence of real-world adversaries.

Building Resilient AI Defenses

Pangea’s report urges enterprises to move beyond default LLM safeguards and adopt a comprehensive, layered defense strategy:

• Multi-Layered Guardrails: Implement protections to block prompt injection, safeguard system prompts, and detect malicious entities using both statistical and AI-driven analysis.
• Attack Surface Reduction: Restrict input languages, operations, and response types in sensitive contexts to minimize exploitable vectors.
• Continuous Security Testing: Regularly conduct red team exercises to probe AI applications for evolving attack techniques.
• Dynamic Temperature Management: Lower model temperature settings in high-security scenarios to reduce output unpredictability.
• Dedicated Security Resources: Assign specialized teams or partner with AI security providers to monitor and respond to new threats.

The full report, “Defending Against Prompt Injection: Insights from 300K attacks in 30 days,” is available from Pangea for organizations seeking to fortify their AI deployments against this rapidly evolving threat.

Find this Story Interesting! Follow us on LinkedIn and X to Get More Instant updates