Red teaming

Red teaming comes from security practice, where a team plays the attacker. For AI, it covers both security attacks (prompt injection, jailbreaks, data extraction, model theft) and harmful behaviour (dangerous instructions, discriminatory content, privacy leaks, hallucinated advice). NIST's Generative AI Profile recommends red teaming with diverse participants, including domain experts and people from affected communities, and the EU AI Act expects providers of general-purpose AI models with systemic risk to perform and document adversarial testing.

Good red teaming has a defined scope, threat model and rules of engagement, logs every finding with severity, and feeds results into mitigations and retesting. It complements, but does not replace, systematic evaluation on benchmarks and ongoing monitoring after deployment. Results are valuable assurance evidence for auditors and boards.

Example: before launching an HR assistant, a company asks testers to coax it into revealing other employees' salaries; they succeed through an indirect question, and access controls on the underlying documents are tightened.