Evaluating and Detecting Hallucinations

Anyone who has used a chatbot for a while has probably noticed that asking the same question twice does not always produce the same answer. The wording may differ a little, the level of detail may shift, or the conclusion itself may change. In a casual conversation this may be harmless. In a task such as scoring work, summarizing a document, or answering a factual question, it becomes a real problem.

This article explains the two main forms of unreliable behavior in language models, inconsistency and hallucination, and walks through the mechanisms that produce them. We will trace how a single prompt can produce different answers, how a small change in wording can change an answer a great deal, and how a model's own generated text can feed its later errors.

The two problems are easy to confuse because both make an answer unreliable, but they are independent. The table below compares them.

Inconsistency also damages trust. In human communication we expect a reasonable level of consistency: a colleague who gave you a different name every time you met would be confusing and hard to rely on. Users bring the same expectation to AI systems, so an answer that shifts from run to run feels unreliable even when each individual answer looks plausible.

Consider the first scenario, in which the same input produces different outputs. Follow one identical prompt through two separate runs. The prompt goes in unchanged both times, but each run produces its own output. The diagram shows the two runs side by side, using the haiku example from the source. Each run is a separate generation, so each can end with a different score.

The source describes the variation in two ways that show up across runs. The wording may differ a little, the level of detail may shift, or the conclusion itself may change. For a scoring task, a change in the conclusion is the most serious of these, because the score is the result the workflow depends on. A workflow that expects a stable result cannot treat these two runs as equivalent.

The second scenario is subtler. Here slightly different inputs produce drastically different outputs. Imagine two prompts that differ by only a few words: Summarize this report, and Summarize this report in one line. The wording change is small, but the request asks for a very different shape of answer, so the output changes sharply. This example is made up for illustration; it is not from the source.

Fixing the generation settings is still useful, but it does not force the model to give the same answer to different inputs. The source is explicit on this point. Settings address variation from the sampling process, while different inputs are different requests. To steer responses toward what you want, the source points to carefully crafted prompts and a memory system. Those tools are covered in later chapters.

A language model produces its output one piece at a time. Each generated token is added to the text the model is working from, and the model then conditions its next choices on that text. One explanation for hallucination is that the model cannot clearly separate the data it was given from the data it generated itself. When it produces a sentence, that sentence becomes part of the context for the next token.

The model then conditions its next choices on that sentence exactly as it would on a fact supplied by the user. If the first generated sentence is slightly wrong, the model can build on it and produce increasingly false claims. Researchers in this area called this pattern self-delusion. Self-delusion describes how a generated claim gets expanded: the model treats its own output as if it were given fact.

Snowballing hallucinations describe a related pattern. After making an incorrect assumption, the model keeps hallucinating in order to justify that original assumption. Each new error supports the earlier one, so the mistake grows larger over the course of the response or conversation. The flowchart shows how each later step is built on the first.

Inconsistency can be traced to randomness in sampling, but the cause of hallucination is more subtle, and sampling alone does not explain it. A model samples from the options it considers probable, so the question becomes how something never seen before can become probable at all.

Training data is far too large to search for every idea, so it is not possible to verify that a generated claim never appeared in training. Researchers therefore rely on hypotheses about why models hallucinate. Two are commonly discussed. The self-delusion view explains how self-supervision can produce hallucinations. The mismatched knowledge view explains how supervision can produce them. Together they suggest that hallucination has several sources, and that a single fix is unlikely to address all of them.

Hallucination is also not a new problem. It was a recognized problem in text generation well before large language models became prominent, and detecting and measuring it has been an established task in natural language generation for years.

It might seem that adding reinforcement learning from human feedback would simply reduce hallucinations. The source reports a more complicated picture. One team found that reinforcement learning helped reduce hallucinations, but a later study found that a model trained with both reinforcement learning and supervised fine-tuning hallucinated more than the same model trained with supervised fine-tuning alone. Human labelers nevertheless preferred the reinforcement-learning model overall, because it improved other qualities.

The source describes several practical measures for reducing inconsistency and hallucination. Fixing sampling settings, caching, prompting, and memory systems all reduce these problems. None of them guarantees perfect consistency or complete prevention of hallucination. The concept map shows how each approach targets the problem and where residual risk remains.

Prompting is one family of approaches. Some users ask the model to answer as truthfully as possible and to say it does not know when it is unsure. This assumes the model has some internal sense of what it knows, which is itself a contested claim, so the instruction helps but cannot be relied on completely. Prompting and context construction techniques covered in later chapters can also help.

Another practical observation is that concise responses may help. The fewer tokens a model must generate, the fewer opportunities it has to invent something. This is a reduction in opportunity, not a guarantee, because a short answer can still be wrong.

Detection is the natural next question. If hallucinations cannot all be prevented, can they at least be detected before users see them? The source notes that detection is not straightforward, because it is hard even for people to detect when another person is lying or making things up. Methods for detecting and measuring hallucinations are covered in a later chapter.