Extracting and Evaluating Model Responses

Many machine learning models are judged with a single, simple calculation. If a model is fine-tuned to assign one of a fixed set of labels, every input has a known correct answer. You run the model over the test inputs, count how many predicted labels match the known answers, and report the proportion of matches. That number is a clean measure of quality. This article explains why the same calculation does not work for an instruction-fine-tuned LLM such as a chatbot, and what evaluators do instead.

A classifier that is fine-tuned to assign labels has a clear output space. Its answer is one of a fixed set of labels, so the output can be checked directly against a known answer for each input. The evaluation question is simply whether the chosen label matches the expected one.

An instruction-fine-tuned LLM works differently. It is asked to follow instructions and produce open-ended responses. The output is free-form text, and there is usually no single correct string to compare against. Two responses can be different in wording, length, and structure and still both be helpful. A string match would mark good answers wrong, so the accuracy calculation cannot be applied directly.

The source names three approaches for evaluating instruction-fine-tuned LLMs. They are not interchangeable. Each one answers a different question about the model, and each one relies on a different kind of scorer.

Short-answer and multiple-choice benchmarks such as MMLU test general knowledge of a model. For multiple-choice items, the answer key is a fixed option, such as B. A chatbot, however, usually returns a full sentence or paragraph. Before the response can be checked against the key, the evaluator has to pull a single answer out of the free text. The diagram shows one way to think about that pipeline.

Human preference comparison compares LLMs based on human preference. The LMSYS chatbot arena is the source's illustration. Instead of matching an answer key, the evaluation shows people two candidate responses to the same prompt and records which one they prefer. Because the question is about preference, a human rater is the scorer.

Each recorded choice is one vote for one of the two responses. Aggregating many such choices gives a win rate: the share of comparisons in which a model's response was preferred. The win rate is a comparison result, not a match against a stored correct string.

Automated conversational benchmarks replace the human judge with another language model. AlpacaEval is the source's example, and it uses another LLM such as GPT-4 to evaluate responses. The sequence below traces the path step by step. Notice that there are two separate LLMs with two separate roles. The model being tested generates the response. The evaluating LLM assesses it.

Read the sequence in order. The prompt goes to the model being tested, which produces an answer. That answer is passed to the judge model, which reads it and returns a judgment. The score comes from the judge, not from a human and not from an answer key. The two LLMs never play the same role: one generates, the other evaluates.

Compare the three approaches by what they need. A multiple-choice benchmark needs a fixed answer key and produces a clear score, but it can only check responses that reduce to one option. A chatbot's conversational quality is open-ended, so there is often no key to compare with. Human preference comparison and automated judges both work without a single correct string. They assess how a response reads and whether it is preferred, which is closer to what conversational performance means.