Retrieval-augmented generation (RAG) systems

Imagine a tool that writes a summary of a long report. The summary reads smoothly, uses correct grammar, and sounds confident. Yet one sentence in it claims something the original report never said. The quality of the writing tells you nothing about whether that sentence is true to the source. This gap between how good a response sounds and whether its claims agree with the facts is the problem this article addresses.

The question a factual consistency evaluation asks is simple to state: does the output agree with what is known, or with the reference it should be based on? Answering it is harder than it sounds, because a model can produce a claim with no obvious signal that anything is wrong. For this reason, evaluators increasingly use AI itself as a judge, using more sophisticated techniques than simply asking a model whether an answer looks right. The two approaches this article focuses on are self-verification and knowledge-augmented verification.

Self-verification starts from a simple intuition. If you ask a model the same question several times and it keeps giving different answers, then at least some of those answers are probably unreliable. A response that the model's other attempts contradict is suspect. The method turns this intuition into a measurement.

SelfCheckGPT is the method the source describes. It rests on an assumption: if a model generates multiple outputs that disagree with one another, the original output is likely hallucinated. To evaluate a response R, the method generates N new responses and then measures how consistent R is with respect to those N responses. The original response is never judged in isolation. Its standing depends on how well the freshly sampled responses support it.

Knowledge-augmented verification changes the source of evidence. Instead of comparing a response with other samples from the same model, it checks individual claims against search results. SAFE, the Search-Augmented Factuality Evaluator, is the method the source describes. It was introduced by Google DeepMind in the paper Long-Form Factuality in Large Language Models, and it verifies a response by leveraging search engine results.

SAFE works in four steps. The pipeline is easiest to understand as a sequence in which a long response is gradually broken down into claims that a search engine can actually test. The diagram traces a response from start to finish and shows where the search engine sits between the claims and the verdict.

Read the pipeline as a chain of hand-offs. The response is decomposed into individual claims. Each claim is then made self-contained. A fact-checking query is proposed for each claim. The search engine returns results that serve as evidence, and a judgment compares each claim with that evidence to produce a verdict. The search engine is the point where outside information enters the process, which is what separates this method from self-verification.

Checking whether a statement is consistent with a given context can also be framed as textual entailment, a long-standing task in natural language processing. Textual entailment is the task of determining the relationship between two statements. One statement is the premise, which here is the context. The other is the hypothesis, which is the output or part of the output being evaluated. The task asks which of three categories the hypothesis falls into.

The three labels map directly onto factual consistency. Entailment implies factual consistency, contradiction implies factual inconsistency, and neutral means consistency cannot be determined. The third label matters because it prevents a forced verdict. When the context simply says nothing about a claim, a neutral result is the honest answer.

Instead of relying on a general-purpose AI judge, you can train scorers that specialize in factual consistency prediction. These scorers take in a pair of premise and hypothesis as input and output one of the predefined classes, such as entailment, contradiction, or neutral. Once the task is framed this way, factual consistency becomes a classification task. The source gives one example, DeBERTa-v3-base-mnli-feveranli, a 184-million-parameter model trained on 764,000 annotated (hypothesis, premise) pairs to predict entailment.

A benchmark gives evaluators a shared test set so that different methods can be compared fairly. TruthfulQA is the benchmark the source names for factual consistency. It comprises 817 questions that some humans would answer incorrectly because of a false belief or misconception. The questions span 38 categories, including health, law, finance, and politics.

What makes these questions hard is that a plausible-sounding answer is often the wrong one. A system that writes fluent text can still produce the false answer that a misconception suggests. TruthfulQA also comes with a specialized AI judge, GPT-judge, which was finetuned to automatically evaluate whether a response is factually consistent with the reference response. The source reports a human expert baseline of 94% on this benchmark. It also shows the performance of several models in a plot taken from the GPT-4 technical report. That plot compares models against one another on the benchmark, so the learner should read it as a comparison between systems rather than as a single score.

Factual consistency is a crucial evaluation criterion for retrieval-augmented generation, or RAG, systems. Given a query, a RAG system retrieves relevant information from external databases to supplement the model's context. The generated response should then be factually consistent with the retrieved context. Retrieval adds a second source of truth, so the question shifts from whether the model knows the answer to whether its answer reflects the material it was handed.

Think of the RAG pipeline as three connected stages: retrieval supplies the reference, generation writes a response from that reference, and factual checking tests whether the response agrees with the reference. Because the checking stage uses the retrieved context as its standard, a failed consistency check means the generated answer is not supported by what the system retrieved. That failure affects the whole system's output, not only one sentence of it.

Neither method is declared the winner in the source. Self-verification is conceptually simple but can be prohibitively expensive. Knowledge-augmented verification depends on the quality of the search results and on the AI judgments made at each step. A practical evaluator chooses based on what evidence is available and what the evaluation can afford.