Extracting Model Responses from Generated Text

Once a model has produced an output for each test instruction, the natural next question is whether those outputs are any good. The practical way to answer it is to place each model response beside the correct answer written for the same instruction and look at the pair. This article follows one such comparison from start to finish.

The comparison begins with a tempting shortcut that does not hold up: the idea that any answer which sounds reasonable must be right. A response is not correct simply because it is related to the right idea. Suppose a question is about thunderstorms and the response names a cloud connected to thunderstorms. At a glance that response can look successful. Checking it against the correct answer shows whether it really does what the instruction asked.

Before any comparison can happen, the model's response has to be separated from the text around it. The generate function returns the input and the output together as one combined string. The model's part is not available on its own, so it has to be pulled out of that string.

The point of this step is to make the comparison fair. If the instruction or surrounding text were left in place, the reader would be comparing the correct answer against a block of text that contains more than the model's answer, and any real difference would be harder to see.

The printed comparison presents the evidence in a fixed order: the instruction text first, then the correct response, then the model response, followed by a separator line. Because the two answers always appear in the same place for every example, your eye can move straight from the correct answer to the model answer and notice any gap.

The diagram shows the flow in order. The model output is first separated into the response alone, then placed in the fixed layout beside the reference, and only then is the verdict formed by looking at where the two diverge.

The test set in this example contains three instructions. Each one has a correct response written for it and a model response to compare. Working through all three shows how different verdicts arise from the same comparison procedure.

When a response is judged wrong, the useful question is where it departs from the expected answer. In the cloud example, the topic is right: both answers concern thunderstorms and both name a cloud. The departure is in the name of the cloud itself. The reference names cumulonimbus, while the model names cumulus.

Reading the pair line by line, the topic lines up and the first difference appears at the cloud name. That single point of divergence is what separates a response that is close from one that is fully right. It also suggests a diagnosis: the model has the subject area right but has chosen a related term where the instruction asked for a specific one.

A response can differ from the reference in wording and still be clearly correct. The simile example shows this. The reference uses lightning and the model uses a bullet, and the words are different. Both still compare the car's speed to something fast, so the task is done. The departure from the reference does not change whether the instruction was fulfilled.

A single verdict describes one instruction. An overall picture comes from combining the verdicts across several instruction, reference, and model response triples. In the three-triple example, one response is clearly correct, one is acceptable despite different wording, and one is close but not entirely accurate.

Read together, the table suggests that this model handles the task type well in the cases where the answer is a recognizable fact or a simple rewrite, but it has a specific weakness where the question requires one precise term among related ones. Three triples is a very small set, so this is a rough picture rather than a reliable measure. Reading the pattern of divergences is more informative than counting correct verdicts alone.