Prompt injection
Definition
An attack that hides instructions in user input or in content an AI model reads, causing it to ignore its intended rules and act for the attacker.
Prompt injection is ranked first in the OWASP Top 10 for LLM Applications. Direct injection happens when a user types instructions designed to override the system prompt. Indirect injection is subtler: the malicious text sits in a web page, email, document or tool response that the model processes, so an assistant summarizing an inbox may obey a hidden line telling it to forward messages. NIST AI 100-2 treats it as a form of adversarial attack on generative systems.
The risk grows with the autonomy and permissions of the system: a chatbot that only answers is less exposed than an agent that can send email, browse or execute code. Mitigations include least privilege, separating trusted instructions from untrusted content, human confirmation before sensitive actions, output filtering, monitoring and red teaming. No current technique eliminates the risk entirely, which is why design and governance matter.
Example: a job applicant hides white text in a CV telling a screening assistant to rate the candidate as excellent.
Where this term is assessed
Assessed in the Responsible AI Foundation (RAIF), domain 2: Risks and harms.
Also covered in: Responsible AI Practitioner (RAIP)
Primary sources
- OWASP Top 10 for LLM Applications (LLM01)
- NIST AI 100-2 (Adversarial Machine Learning)