Developing and Validating AI Models for Claims Summarization
Validation of the AI model used four separate criteria: factual accuracy, citation correctness, tone, and completeness, and each was judged on its own.
Why One Headline Score Can Mislead
Imagine a model that writes claim narratives and is reported as mostly accurate. That description sounds reassuring, but it leaves an important question unanswered: accurate about what, and for whom? A narrative can state the right facts and still be written in a way that damages the business. It can also be fluent and persuasive while leaving out information the staff needed. Validation exists to separate these concerns so that one strong result cannot hide a weak one.
This article follows a single validation effort on a claims-processing model. It shows which checks were used, what the first run revealed, how the team responded, and why the final result was judged on its own terms rather than by a simple average.
Four Questions Asked of One Narrative
The model's output was validated against four separate criteria: factual accuracy, citation correctness, tone, and completeness. Each criterion asks a different question about the same narrative, and each was judged on its own.
Factual accuracy asks whether the statements in the narrative are true.
Citation correctness asks whether the sources attached to those statements actually support them.
Tone asks whether the narrative's stance and manner are appropriate for the business context.
Completeness asks whether the narrative includes the information it should have included.
| Criterion | Question it asks | Failure it catches |
|---|---|---|
| Factual accuracy | Are the statements true? | False statements |
| Citation correctness | Do the attached sources support the statements? | Statements whose sources do not back them up |
| Tone | Is the stance appropriate for the business context? | A slanted narrative, even when every sentence is defensible |
| Completeness | Does the narrative include what it should? | Omitted information |
The four validation criteria, each examining a different aspect of the same claims narrative
The four questions are independent. A narrative can be factually accurate and well cited while still being slanted, or it can be balanced in tone and still be missing fields. This independence is the reason the team validated against all four rather than relying on one headline figure.
Tracing a Baseline Validation Run
The first validation run is called the baseline. In this run the model cleared two criteria, factual accuracy and citation correctness, but missed the tone and completeness criteria. The source does not report the individual scores behind these outcomes, so the useful information is which criteria passed and which did not.
Follow one narrative through the checks. Each criterion reads the same narrative and returns its own result. Two results are passes and two are misses. Those four results then feed a single overall check, which did not pass. The diagram below traces that path.
Why Passing Some Criteria Is Not Enough
The baseline result shows why the criteria must be read separately. The two passing criteria were the ones about the content's correctness and sourcing. The two failing criteria were about the narrative's judgment and coverage. A reviewer who looked only at whether the facts were right would have seen a model that appeared ready. The fuller picture emerged only because each criterion was measured on its own.
Mistakes When Reading a Validation Result
Common mistake Treating a summary of passes as an overall pass
What it looks likeA report says two of four criteria cleared, so the model is mostly ready.
Why it failsThe overall check did not pass in the baseline run, even though two criteria cleared. Partial success on some criteria does not establish readiness.
Do this insteadRead each criterion's result separately and confirm what the overall check actually requires.
Common mistake Assuming a narrative with true facts has no tone problem
What it looks likeEvery factual statement checks out, so the narrative must be fine.
Why it failsThe tone failure in the source involved no false statement. The overall impression was slanted even though each sentence was defensible.
Do this insteadRun a separate tone check that asks about stance and manner, not only about correctness.
Common mistake Blaming the instructions for every missing field
What it looks likeThe model left out a field, so the prompt must be wrong.
Why it failsThe source describes a completeness gap caused by missing inputs, because the model did not receive data that processors could see.
Do this insteadCheck whether the model was given the information before changing the instructions.
Before accepting a pass, confirm what the threshold is, whether it applies to each criterion or to a combined score, and whether the new figures are being compared with the earlier ones on equal terms.
Closing the Gaps with Targeted Revisions
Once the gaps were identified, the team made changes aimed at each cause rather than one general adjustment. The source reports that two prompt revisions and one schema change closed both gaps. A prompt revision changes the instructions the model receives. A schema change alters how the information is structured for the model. Each kind of change maps to a different kind of problem.
Matching each revision to its cause
The baseline run missed tone and completeness for different reasons. Decide which kind of revision addresses each failure.
- 1Identify the tone cause
The narratives were slanted toward the claimant while every sentence stayed defensible. This is a problem in how the model was instructed to write, so it calls for a change to the instructions.
- 2Identify the completeness cause
Processors could see fields the model never received. This is a problem in how information reached the model, so it calls for a change to how the data is structured.
- 3Match each change to its cause
Prompt revisions address the tone problem, and the schema change addresses the completeness problem. The source's count of two prompt revisions and one schema change fits this split, though it does not say exactly which revision did which job.
Each gap was closed by a revision matched to its cause, rather than by a single broad change.
Re-validation then compares the revised run against the baseline on the same four criteria. The diagram below shows how each criterion's status is compared across the two runs.
Judging the Final Result on Its Own Terms
The retest run reached 76 percent across all four criteria, and the source states that the threshold was cleared. The customer then signed the validation milestone payment. The payment followed the validation result, which is why the result has to be documented and defensible. A result that is reported only as an average could hide a weak criterion, which is exactly the risk the four separate criteria were designed to prevent.
Checking a Validation Report
What do you think happens?
A narrative passes factual accuracy and citation correctness but fails tone. Does the model pass the overall check?
Reveal answer
No, the overall check does not pass on that basis.
The criteria are independent, and the baseline run shows a model clearing two criteria while missing others and not passing overall. Passing criteria do not cancel out a failing one.
A validation report lists a model's results as passing factual accuracy and citation correctness, with tone and completeness both missed. A colleague proposes reporting only the number of criteria that passed. Explain why this reporting choice could mislead readers, and name what the report should show instead.
Hints
- Consider whether a count of passes reveals which criteria failed.
- The source says a single overall figure can mislead.
- Think about whether each criterion's result should be visible on its own.
Key Takeaways
- Validation of the claims-narrative model used four separate criteria: factual accuracy, citation correctness, tone, and completeness, each judged on its own.
- A model can clear some criteria and still fail the overall check, because a single overall figure can hide a weak criterion.
- Tone and completeness failures had different causes, a slanted narrative and omitted fields, so each was addressed by a matched revision.
- Two prompt revisions and one schema change closed the gaps, and the retest reached 76 percent across all four criteria and cleared the threshold.