Writing and Revising Evaluation Rubrics

Imagine a system that drafts clinical notes for physicians to review. Its quality is tracked on a dashboard, and the number that matters most is first-pass acceptance: the share of drafts a clinician signs off on without rewriting them. When that number dips, the natural question is whether the drop is real and, if it is real, which part of the pipeline is responsible.

This article follows one such dip from detection through repair. The repair involved two separate changes that had to be made in the right order, and the reasons for that order are the point of the lesson. Keep one question in mind as you read: if the scoring system is still giving reasonable-looking numbers, how would you know it has stopped measuring the thing you care about?

A single aggregate number summarizes many outputs at once. That summary is useful, but it can look acceptable while one subgroup quietly disagrees with the criteria used to judge it. The average is a blend of every population the system serves, so a drop inside one population can be diluted by the others. A flat overall line does not prove that every group is being served well.

The repair began with a labeling session. An orthopedist reviewed forty cases over two hours and gave feedback specific enough to act on. Three points came out of that session. First, the physical-exam section had to be organized by joint rather than by type of finding. Second, the assessment paragraph had to follow the structure of a primary diagnosis, then a differential, then a plan, which is the structure she had been trained to use. Third, the patient-history section could be far shorter than the template used for family-practice notes.

Notice what this chain does not include. Nothing in the account says the team rewrote its scoring method or changed the dashboard. The repair was driven by a domain expert's judgment about what a good orthopedic note looks like, and that judgment was then written into the instructions the model follows.

It is tempting to treat a regression fix as a single action, but the repair contains two distinct changes that answer two distinct questions. The first change asks what the generator should be told to produce. This is the prompt revision, which encodes the specialist's standard into the instructions. The second asks whether the thing grading the generator's output still understands what good output looks like for the new population. This is the judge recalibration, performed on a fresh sample from the affected population.

Read the diagram as a dependency chain. The specialist's feedback is the raw material for the prompt revision. The revised prompt produces new outputs, and those outputs form the fresh sample. The judge is then recalibrated on that sample, so its grading reflects the new population. Only after both changes are in place can the team confirm that the regression is actually repaired.

Because the feedback was concrete, the prompt revision itself was quick: it took an afternoon, and the fix shipped on a Friday. The result showed up in the following week's dashboard, where first-pass acceptance had recovered to sixty-three percent. The orthopedic cohort, which had been lagging, began to track in line with the family-practice cohort for the first time.

The diagram above is an illustrative generated view, not a reproduction of the source dashboard. It shows the pattern the source describes: an aggregate that can remain acceptable while one cohort lags, and a per-population breakdown that makes the lag visible and then shows it closing after the revision.

An inline judge is the component that grades outputs as the system runs. The source makes an important distinction: an inline judge can remain useful for flagging outputs on a new population even after its absolute scores have become unreliable. The flag and the score must therefore be judged separately, because they support different claims.

The deeper lesson is about the rubric, the set of criteria that defines a good output. The source makes the point through the criteria-drift work of Shankar, and its statement is blunt: the rubric is not done. A rubric encodes what good looks like for a particular population, and it has to be extended whenever the system meets a population whose definition of good has not yet been written into it.

The loop has no natural end. Each time the system meets a population whose definition of good has not been written down, the cycle starts again. In the clinical notes case, the orthopedic cohort held a different definition of good, and the per-trace loop is what made that difference impossible to miss.