What is an AI hallucination?

What it means at work

A hallucination is an AI output that is false but delivered with complete confidence. A language model writes text that is statistically likely to follow your request. Likely-sounding is not the same as true, so sometimes the most plausible-sounding answer is simply wrong. On this point, the Anthropic documentation says that even the most advanced models can sometimes generate text that is factually incorrect.

At work, hallucinations show up as:

• Invented numbers: a growth rate or market size that appears in no report.
• Fake sources: citations with realistic titles, authors and links that do not exist.
• Wrong details: a misdated event, a misattributed quote, a policy that was never written.
• Spreadsheet errors: a formula that points at the wrong column but looks correct.
• Summaries that add: a document summary that includes a point the document never made.

Why it happens

The model is not looking facts up. It is generating. That makes hallucinations more likely when:

• the question needs specific facts you did not provide, such as a figure, a name or a recent event;
• you ask for sources, quotes or numbers on obscure topics;
• the prompt pushes it to produce an answer rather than admit uncertainty;
• the conversation or attached file is very long and details get lost (see context window).

In the paper, Kalai et al. write that large language models sometimes guess when uncertain. They argue that training and evaluation procedures reward guessing over acknowledging uncertainty.

An example

An analyst asks for market-size figures “with sources.” The tool returns tidy numbers and two links that lead nowhere. The fix is procedural, not technical: give it the documents to draw from, ask it to quote the passage behind each claim, and open the sources before anything goes into a deck.

How to catch and reduce them

1. Ground it. Paste or attach the source material, so the AI works from your text instead of memory. Tools that retrieve documents first use RAG for this reason.
2. Allow “I don’t know.” The Anthropic guidance is to explicitly give the model permission to admit uncertainty. A line such as “If the answer is not in the document, say ‘not stated’” does this.
3. Ask for evidence. For long documents, Anthropic recommends asking the model to extract word-for-word quotes first before performing its task. Then check a sample.
4. Verify what matters. Names, numbers, dates, quotes and legal or financial claims get checked against the original. Anthropic’s example prompt tells the model to find a direct quote for each claim and remove claims it cannot support.
5. Use search-connected tools for current facts. A tool that shows live sources, such as Perplexity, lets you click through and confirm.

The longer routine is in how to stop AI hallucinations, and summarizing PDFs and contracts with AI shows it on real documents.

What the sources say

• In the paper, Kalai et al. say hallucinations persist even in state-of-the-art systems.
• In the paper, Kalai et al. propose modifying the scoring of existing benchmarks that are misaligned but dominate leaderboards.
• The Survey of Hallucination in Natural Language Generation says deep learning based generation is prone to hallucinate unintended text.
• Liu et al. found performance significantly degrades when models must access relevant information in the middle of long contexts, which is why details in long attachments get missed.

A common misconception

Some believe newer models never hallucinate, or that a confident tone signals accuracy. Neither holds. Better grounding helps, but no model is free of them, and a model can be fluent and wrong in the same sentence. Verification stays part of the job.

Spotting these errors is a habit, and it is easiest to build by practicing on sample documents. That is a core part of the hands-on work in Learn AI Fast, a two-day in-person class in Boston for non-technical professionals.

Sources

• Anthropic, Reduce hallucinations: “can sometimes generate text that is factually incorrect or inconsistent with the given context”
• Kalai et al., “Why Language Models Hallucinate” (arXiv): “large language models sometimes guess when uncertain”
• Kalai et al., “Why Language Models Hallucinate” (arXiv): “training and evaluation procedures reward guessing over acknowledging uncertainty”
• Anthropic, Reduce hallucinations: “Explicitly give Claude permission to admit uncertainty”
• Anthropic, Reduce hallucinations: “ask Claude to extract word-for-word quotes first before performing its task”
• Anthropic, Reduce hallucinations: “find a direct quote from the documents that supports it”
• Kalai et al., “Why Language Models Hallucinate” (arXiv): “persist even in state-of-the-art systems”
• Kalai et al., “Why Language Models Hallucinate” (arXiv): “modifying the scoring of existing benchmarks that are misaligned but dominate leaderboards”
• Ji et al., “Survey of Hallucination in Natural Language Generation” (arXiv): “deep learning based generation is prone to hallucinate unintended text”
• Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (arXiv): “significantly degrades when models must access relevant information in the middle of long contexts”