Implementing Automated Evaluation: Scoring Model Responses

Once a model has been instruction-tuned, the next question is how well it follows instructions on examples it has not seen during training. A common approach is to take a held-out test set. Each entry in that set already contains an instruction, an optional input, and a reference output. The model is asked to answer every instruction, and those answers are stored alongside the original entries so they can be compared later.

This article follows that practical loop from start to finish: generating a response for each entry, writing it into the dataset, saving the result to a file, and confirming that the response landed where it should. Before you read on, think about what a single pass through the loop has to accomplish. The loop starts with an entry that holds an instruction, an input, and a reference output. It has to finish with a model answer attached to that same entry. Keep that goal in mind as you trace the steps.

The loop reuses the same generate method that was used earlier in the chapter. The difference is that it now runs once for each entry in the test set instead of once for a single example. The loop is driven by enumerate over test_data, so each iteration has both an index i and the entry itself. A tqdm progress bar wraps the loop, which is why the run reports how many of the 110 entries have been processed.

For each entry, the code first builds a prompt from that entry using format_input. That prompt is converted into token IDs with text_to_token_ids and moved to the target device. Those IDs are passed to generate, which is told to produce at most 256 new tokens, to use the context length stored in BASE_CONFIG, and to stop when it reaches the end-of-sequence token whose ID is 50256. The output of generate is a sequence of token IDs that contains the prompt followed by the newly generated continuation.

Notice the shape of the loop. It visits each entry in order, performs the same sequence of steps for each one, and then moves on. The decision at the end of the body is what repeats the work until every entry is finished. The diagram above shows that order for a single entry.

The decoded string is not yet the answer. The code converts the whole token sequence back into text with token_ids_to_text, so the decoded string still begins with the prompt. To isolate the answer, the code performs three operations. First, it slices off the first part of the string whose length equals the length of the prompt text. Second, it removes the Response marker wherever it appears in the remaining text. The marker is the line made of three hash signs, the word Response, and a colon. Third, it calls strip to trim surrounding whitespace. What remains is the response text alone.

The response is not returned from the loop or collected in a separate list. Instead, the code writes it directly into the entry it came from, using the index i to pick the right dictionary inside test_data. Because the dictionary is modified in place, the change is visible everywhere test_data is used afterward, including the file-writing step. Each cleaned response is stored under the key model_response, which keeps it paired with the original instruction, input, and output.

The file is not written inside the loop. The write step comes after the last entry has been processed, so the complete dataset, with every model_response filled in, is serialized in one operation. After the loop finishes, the updated test_data is written to a file named instruction-data-with-response.json. The file is opened in write mode 'w', and json.dump serializes the list into that file using an indent of 4 spaces, which makes the saved structure easy to read.

Because the write happens once, after the loop, the file does not exist in a partial state during the generation work. The data is accumulated in memory in test_data as the loop runs, and the disk is touched a single time when everything is ready.

Each entry in the saved file keeps its instruction, input, and output, and gains the new model_response field. This makes the file a complete record of the test run rather than a bare list of answers. The structure is a list of entries, where each entry is a set of named fields. The diagram below shows one entry in that structure.

Once the run has finished, it is worth inspecting one entry directly rather than trusting that the loop worked. Look at a single entry from the updated test set. It should contain the original instruction, the input, the reference output, and a model_response field whose value is the model's answer. Seeing all four keys together confirms that the response was added to the correct dictionary and that the original fields were not disturbed.