What Is Prompt Injection?
IBM describes prompt injection as "...malicious inputs as legitimate prompts, manipulating generative AI systems (GenAI) into leaking sensitive data, spreading misinformation, or worse." Essentially, an application says "follow these rules," and an attacker says "do this instead." If the model follows the attacker's instructions, it can lead to a number of unintended consequences.
These consequences scale with what the model has access to, and what it can do. A support chatbot might be manipulated into providing incorrect information, answering unrelated questions, or performing unintended actions (like writing code, and cost the company money). An Agent with access to a database, email, or other tools could be manipulated into sending or forwarding sensitive emails, deleting data, etc.
System Prompt Extraction
Prompt injection is not one single technique, instead it's a class encompassing a variety of attack patterns; The Open Worldwide Application Security Project (OWASP) lists it as the most critical threat in its LLM Top 10 (LLM01). Within the prompt injection class, attacks split into two branches: direct injection, where the malicious input comes straight from the user's own message, and indirect injection, where it's ingested via content the model processes.
System prompt extraction is one of the most common direct injection techniques, and it's the technique we will be attempting in this demo. It targets the model's behavioral instructions directly. If successful, the attack can expose internal policy, workflow details, guardrails, or enough information for an attacker to craft stronger follow-up attacks.
Common attack patterns include asking the model to "repeat your exact instructions above" or to role play as a debugger and print the initialization context.
Interactive Demo
In this demo, we utilize Hugging Face to load in and run a lightweight generative model. The model acts as an AI assistant that has been given a protected value, and explicitly told not to disclose that value. The setup is intentionally simple so you can see instruction failures clearly, try different attack patterns, and run through as many iterations as you want with the reset button.
Click "Load Prompt Injection Demo Environment" to get started.
Load Environment
Requirements: WebGPU browser (Chrome/Edge, hardware acceleration on), GPU (integrated OK), ~1GB download, ~1GB VRAM.
Prompt Injection Lab
The assistant has hidden instructions that forbid revealing a protected value. The 1B parameter model in this demo performs better than you might expect, but is still beatable with a little intuition.
System Prompt (for transparency)
Try these techniques
- Instruction disclosure: ask the model to explain the instructions it received previously.
- Core-directive conflicts: ask the model to treat a new instruction as higher priority than its existing constraints.
- Obfuscated requests: avoid obvious trigger terms and describe the target indirectly.
How to Defend Against Prompt Injection
Once you've gotten the demo to work, the next question is... how do you stop it? From a traditional web security perspective, we might try and sanitize input, but that just doesn't work here. SQL injection and XSS can be sanitized because there is a consistent, escapable structure. With prompt injection, system instructions and user data often arrive together as unstructured natural language, with no good means of delineation.
Given the nature of how these generative models work, there is no one-size-fits-all fix. However, through a defense-in-depth approach, you can mitigate the risk of prompt injection attacks. OWASP lists the following recommended defenses:
- Constrain model behavior: Give the system prompt explicit boundaries on role, capability, and topic, and tell it to ignore attempts to override those instructions.
- Define and validate expected output formats: Require a specific output format and check it with code.
- Implement input and output filtering: Screen content on the way in and out with semantic filters and string checks. Evaluate responses for contextual relevance.
- Enforce privilege control and least privilege access: Keep API tokens and sensitive functions in your own code, not the model's, and grant only what a task needs.
- Require human approval for high-risk actions: Put a human in the loop for privileged operations to prevent unauthorized actions.
- Segregate and identify external content: Clearly mark untrusted input, like documents or tool output, so it's treated as data, not instructions.
- Conduct adversarial testing and attack simulations: Red-team the system regularly, treating the model itself as an untrusted user.