Reducing Hallucinations

Why models make things up

A hallucination is confident, plausible output that is false or invented. The US National Institute of Standards and Technology (NIST) prefers the word confabulation: the production of confidently stated but false content by which users may be misled. NIST lists it as one of twelve risks specific to generative AI in its 2024 Generative AI Profile (NIST AI 600-1). The word fits well: like a person who confabulates, the model is not lying; it fills a gap with something that sounds right.

The mechanism is explained on Limits and Myths: a model picks a likely next token given everything before it, and a sentence with the right shape and vocabulary gets a high probability whether or not it is true. There is no separate "lookup" step that could fail visibly. OpenAI researchers argued in 2025 that training and evaluation also reward guessing: a test that counts only right answers scores a confident guess higher than an honest "I don't know", so models learn to guess.

For prompting, the practical question is where the gaps are. The risk is highest when the answer depends on:

• Your own organization. Kestrel Energy's knowledge articles, runbooks, change numbers and people are not in any model's training data.
• Exact details. Version numbers, command options, error codes, prices, dates, quotes and references. The model knows what such a detail looks like, which is exactly what makes an invented one convincing.
• Recent events after the model's training cut-off, unless the assistant searches the web or you supply the information.
• Rare topics that appear only a few times in public text, such as an old firmware or a niche protocol.
• A false premise in the question. "Why does KB-0999 say to disable the firewall?" invites an explanation of an article that does not exist.

Ground the answer in sources

The most effective single step is to put the facts in the prompt, so the model does not have to remember or invent them (see Giving Context and Reference Material for how to include and mark off documents). Including them is not enough on its own: the model may still mix in its general knowledge. A grounding instruction tells the model explicitly to base its answer only on the material provided and not on what it learned in training. Anthropic's guidance recommends exactly this restriction against hallucinations.

What should a home worker do when the Kestrel Energy VPN shows error 809?

<article>
KB-0217 Home-worker VPN: error 809.
Error 809 means the network blocks VPN traffic.
Switch to a phone hotspot, or ask the venue to allow VPN.
If the error persists on a home network, contact the service desk.
</article>

Using only the knowledge article above, not general knowledge, tell a home worker what to do about VPN error 809. If the article does not cover their situation, say so.

The first prompt will usually get a sensible-sounding list of generic VPN fixes, some of which do not apply to Kestrel's setup. The second limits the answer to the article. Applications that retrieve the right documents automatically and put them in the prompt use the same principle at scale; that is grounded generation with retrieval, and its limits are the same: if the retrieved documents are wrong or missing, a grounded answer is wrong too.

Allow "I don't know"

A model tends to answer whatever you ask; the shape of a question pulls an answer out of it. An abstention is the model explicitly declining to answer, saying it does not know or that the source does not cover the question, instead of guessing. You make abstention more likely by offering it as a legitimate result, with exact wording:

Answer the question using only the procedure in the <procedure> tags. If the procedure does not answer the question, reply exactly: "The procedure does not cover this." Do not fill gaps from general knowledge.

Anthropic writes that explicitly permitting the model to admit uncertainty can drastically reduce false information. Microsoft makes a related point from the other side: a bare instruction such as "only write true facts" or "don't make things up" may not be an adequate mitigation, because the model has no way to tell which of its facts are true. What helps is giving it a concrete, allowed alternative ("reply 'not found'") and the material to decide with.

A fixed abstention phrase has a second benefit: a program can recognize it. An application can route "The procedure does not cover this." to a person, while a free-form "I'm not entirely sure, but…" slips through as an answer.

Ask for quotes before the answer

For long documents, Anthropic recommends asking the model to first extract the word-for-word quotes that are relevant to the question, and then to answer based on those quotes (the technique is introduced on Giving Context and Reference Material). This helps in two ways. The model has to commit to concrete evidence before it writes conclusions, and you get the evidence in a form you can check.

<runbook>
…(the full runbook text)…
</runbook>

Question: may the NOC restart the edge firewall during business hours?

First, copy the sentences from the runbook that are relevant to this question, word for word, inside <quotes> tags. If there are none, write "no relevant text". Then answer the question in at most two sentences, using only those quotes.

Checking that a quote really appears in the source is mechanical, so let a program do it. The example below takes a recorded answer (real answers differ from run to run and between models) and looks up every quoted sentence in the source text.

import re

# The source the model was given (a fictional knowledge article)
source = """KB-0217 Home-worker VPN: error 809.
Error 809 means the network blocks VPN traffic.
Switch to a phone hotspot, or ask the venue to allow VPN.
If the error persists on a home network, contact the service desk."""

# A recorded answer that was asked to quote its evidence in "double quotes"
answer = (
    'KB-0217 covers this. It says "Error 809 means the network blocks VPN traffic." '
    'and "Switch to a phone hotspot, or ask the venue to allow VPN." '
    'It also says "Restart the laptop before you try again."'
)


def normalize(text):
    """Ignore differences in spacing and letter case, nothing else."""
    return " ".join(text.split()).lower()


quotes = re.findall(r'"([^"]+)"', answer)
for quote in quotes:
    found = normalize(quote) in normalize(source)
    print("ok      " if found else "MISSING ", quote)

ok       Error 809 means the network blocks VPN traffic.
ok       Switch to a phone hotspot, or ask the venue to allow VPN.
MISSING  Restart the laptop before you try again.

The third "quote" reads like the other two but appears nowhere in the article: an invented quote, flagged in milliseconds. The check ignores only spacing and letter case; anything looser (accepting "close enough" quotes) would let paraphrases with a changed meaning through. A quote that passes proves only that the sentence exists, not that it supports the conclusion drawn from it. That part stays human work.

Citations and citation checks

When the model works with several sources, number them in the prompt and ask for a citation after every sentence or claim: "Cite the source number in square brackets after each sentence, like [2]. Leave out any claim you cannot cite." Microsoft notes that with citations, a wrong answer requires the model to make two errors, a wrong claim and a wrong reference that fits it, and that citations placed inline next to each claim work better than a list of sources at the end.

A citation check verifies the references in an answer: that every claim has one, that each cited source exists, and ideally that the source actually supports the claim. The first two parts are easy to automate:

import re

# Numbered sources that were in the prompt
sources = {
    1: "KB-0217 Home-worker VPN: error 809",
    2: "KB-0230 Multi-factor sign-in on new phones",
    3: "KB-0244 Portal password reset",
}

# A recorded answer that was told to cite sources as [n] after each sentence
answer = """Error 809 means the network blocks VPN traffic [1].
A phone hotspot usually works as a workaround [1].
Users on a new phone must register the authenticator app again [2].
The VPN client must be version 6.2 or later [4].
Restarting the router fixes most cases."""

for sentence in answer.splitlines():
    cited = [int(n) for n in re.findall(r"\[(\d+)\]", sentence)]
    if not cited:
        status = "no citation"
    elif all(n in sources for n in cited):
        status = "cites " + ", ".join(f"[{n}]" for n in cited)
    else:
        status = "UNKNOWN SOURCE " + ", ".join(f"[{n}]" for n in cited if n not in sources)
    print(f"{status:<18} | {sentence}")

cites [1]          | Error 809 means the network blocks VPN traffic [1].
cites [1]          | A phone hotspot usually works as a workaround [1].
cites [2]          | Users on a new phone must register the authenticator app again [2].
UNKNOWN SOURCE [4] | The VPN client must be version 6.2 or later [4].
no citation        | Restarting the router fixes most cases.

Two lines are caught: a citation to a source that was never provided, and a claim with no citation at all. Two problems are not caught, and could not be by a simple program. "A phone hotspot usually works" cites a real source that does not say "usually". And the line about new phones cites source 2, which is the right topic, but the program cannot tell whether the article actually says that. Automated checks remove the obvious failures quickly; supporting evidence still needs a reader, or a second prompt that compares each sentence with its cited source.

Fabricated references

A fabricated reference is an identifier, link or source in a model's answer that looks real but does not exist: a knowledge article number, a URL, an RFC or paper, a ticket, a command option, a configuration setting or a software package. They are particularly convincing because they follow the right pattern. Ask a model which Kestrel Energy article covers error 809 without giving it the articles, and "KB-0412" sounds as plausible as the real KB-0217.

• Links and documents: open them. A URL that returns "page not found" or leads somewhere else is the most common case.
• Commands and options: check the manual page or official documentation before running anything (see Reading Errors, Logs and Documentation).
• Software packages: a suggested package that does not exist is not just an error but a security risk, because attackers register such names; see package hallucination.
• Your own identifiers: look up every ticket, change and article number in your own systems.

Verify the answer

Techniques in the prompt lower the rate of hallucinations. Checks after the answer catch the ones that remain. From quick to thorough:

• Ask twice and compare. Anthropic suggests generating several answers to the same prompt and comparing them: inconsistencies between the outputs can indicate hallucinations. A fact that changes between runs deserves suspicion; the majority vote version of this idea is self-consistency.
• A separate review step. A second prompt that checks each claim against the source, in a new request, is more effective than "double-check yourself" at the end of the first (see draft, review, refine).
• Automatic checks for what is mechanical: quotes, citations, identifiers, the shape of structured output.
• A person for what matters: anything sent to customers, anything that changes a system, anything that is decided on. How much checking an answer needs depends on what happens if it is wrong; see Using AI at Work.

Which Kestrel Energy knowledge article covers VPN error 809 for home workers? Give the article number and what it says.

Here are three Kestrel Energy knowledge article summaries.

<article id="KB-0217">Home-worker VPN: error 809. Error 809 means the network blocks VPN traffic. Switch to a phone hotspot, or ask the venue to allow VPN.</article>
<article id="KB-0230">Multi-factor sign-in on a new phone: register the authenticator app again via the self-service portal.</article>
<article id="KB-0244">Portal password reset: use "Forgot password" on the login page; the link is valid for 30 minutes.</article>

Which of these articles covers VPN error 809? Give the article id and quote the sentence that says so, word for word. If none of the articles covers it, answer only "none".

What prompting cannot fix

Every technique on this page lowers the rate of invented content; none brings it to zero. Anthropic's own guidance says the same: the techniques reduce hallucinations but do not eliminate them, so critical information must always be validated. The OWASP Top 10 for Large Language Model Applications lists the risk as Misinformation (LLM09 in the 2025 edition, LLM07 in the 2026 edition), with people's overreliance on the output as a central part of the problem.

That human side deserves as much attention as the prompt. A fluent, well-formatted answer is easy to trust, and the more often a tool is right, the less carefully its answers get checked (see overreliance and automation bias). Keep the checks proportional to the stakes, and keep them even when the tool has been right for weeks. To measure whether a change to a prompt actually reduces errors, rather than hoping it does, test it on a fixed set of cases; see Iterating and Testing Prompts.