What is Prompt Injection?

Prompt injection is an attack where malicious inputs manipulate large language models into ignoring their instructions, executing unintended actions, or revealing sensitive information.

What is prompt injection?

Prompt injection is an attack technique where adversaries craft inputs that cause large language models to override their system instructions, bypass safety guardrails, or execute unintended actions. It exploits the inability of LLMs to reliably distinguish between trusted instructions and untrusted user input within the same context.

What is the difference between direct and indirect prompt injection?

Direct prompt injection occurs when users type malicious instructions into the LLM interface. Indirect prompt injection embeds malicious instructions in external content the LLM processes, such as web pages, documents, or emails. Indirect injection is more dangerous as it can attack users who never see the injected content.

What are real-world examples of prompt injection?

Real-world examples include manipulating AI chatbots to reveal system prompts, tricking AI assistants into exfiltrating conversation data via generated URLs, embedding instructions in web pages that hijack AI browsing agents, and inserting invisible prompts in documents that alter AI-generated summaries or analyses.

How do you prevent prompt injection?

Prevention strategies include separating system instructions from user input architecturally, implementing input and output filtering, using classifiers to detect injection attempts, applying least-privilege principles to LLM tool access, validating LLM outputs before execution, sandboxing LLM operations, and maintaining human-in-the-loop for sensitive actions.

Why is prompt injection difficult to solve?

Prompt injection is fundamentally challenging because LLMs process instructions and data in the same token space without a reliable separation mechanism. Unlike SQL injection where parameterized queries solve the problem, no equivalent architectural fix exists for LLMs. Defenses remain probabilistic rather than deterministic.

How does prompt injection affect AI agents?

AI agents with tool-calling capabilities face amplified prompt injection risks because injected instructions can trigger real-world actions like sending emails, modifying databases, executing code, or accessing APIs. The attack surface expands with each tool the agent can access, making least-privilege design critical.

What is jailbreaking versus prompt injection?

Jailbreaking attempts to bypass an LLM's safety training to generate harmful content, often using social engineering techniques against the model itself. Prompt injection aims to override the application's system instructions to alter the LLM's behavior within a specific deployment. Both exploit LLM instruction-following but target different boundaries.

How should organizations test for prompt injection?

Organizations should conduct red team exercises using known injection techniques like instruction override, context manipulation, and encoding tricks. Test with indirect injection via documents and web content. Evaluate output filtering effectiveness, tool-call validation, and data exfiltration paths. Use established frameworks like OWASP LLM Top 10.