Hallucination

Hallucinations happen because large language models generate the statistically likely next words, not verified facts. NIST's Generative AI Profile (AI 600-1) prefers the term confabulation and lists it among the risks unique to or worsened by generative AI, alongside information integrity and harmful bias. The risk is highest when users treat fluent text as authoritative: invented case law in legal filings, wrong dosages, nonexistent references or fabricated policy terms given to customers.

Mitigations combine technical and organizational measures: grounding answers in approved sources (retrieval-augmented generation), asking the model to cite and then checking citations, evaluation and red teaming before release, restricting high-stakes uses, clear user warnings and, above all, human review where errors matter. An acceptable use policy should tell staff which outputs must be verified before use. When a hallucination causes harm, it should be handled as an AI incident.

Example: an airline chatbot invents a refund rule; the company is held to the answer it gave the customer.