Text Generation with a Trained GPT Model

Once a model has been instruction-tuned, the next question is how well it follows instructions on examples it has not been trained on. A common approach is to take a held-out test set. Each entry in that set already contains an instruction, an optional input, and a reference output. The model is asked to answer every instruction, and the answers it produces are stored alongside the original entries so they can be compared later.

This article follows the practical loop that does that work. It generates a response for each entry, writes that response into the dataset, saves the result to a file, and then confirms that everything landed where it should. The goal of a single pass is simple to state: start with an entry that holds an instruction, an input, and a reference output, and finish with a model answer attached to that same entry.

The loop reuses the same generate method that was used earlier in the chapter. The difference is that it now runs once for each entry in the test set instead of once for a single example. The loop is driven by enumerate over test_data, so each iteration has both an index i and the entry itself. A tqdm progress bar wraps the loop, which is why the run reports how many of the 110 entries have been processed.

For one entry, the first step is to build a prompt from that entry using format_input. The prompt is then converted into token IDs with text_to_token_ids and moved to the target device. Those IDs are passed to generate, which is told to produce at most 256 new tokens, to use the context length stored in BASE_CONFIG, and to stop when it reaches the end-of-sequence token whose ID is 50256. The output of generate is a sequence of token IDs that contains the prompt followed by the newly generated continuation.

The sequence shows the order of operations for one entry, and the decision at the end repeats the body until every entry is finished. Each iteration does the same steps in the same order. What changes from one iteration to the next is only the entry being processed.

The code converts the whole sequence of token IDs back into text with token_ids_to_text. Because the sequence began with the prompt, the decoded string still starts with the prompt. The answer is the part that follows it. To isolate the answer, the code does three things: it slices off the first part of the string whose length equals the length of the prompt text, it removes the marker '### Response:' wherever it appears, and it calls strip to trim surrounding whitespace. What remains is the response text alone.

The response is not returned from the loop or collected in a separate list. The code writes it directly into the entry it came from, using the index i to pick the right dictionary inside test_data. Because the dictionary is modified in place, the change is visible everywhere test_data is used afterward, including the file-writing step. This is why the response stays paired with its original instruction, input, and reference output.

The three original fields are not replaced. The new key model_response is added beside them, so the same dictionary now carries the question, the reference, and the model's answer together.

After the loop finishes, the updated test_data is written to a file named instruction-data-with-response.json. The file is opened in write mode 'w', and json.dump serializes the list into that file with an indent of 4 spaces. The indentation makes the saved structure easy to read. Each entry in the file keeps its instruction, input, output, and the new model_response field, so the file is a complete record of the test run rather than a bare list of answers.

Once the run has finished, it is worth inspecting one entry directly rather than trusting that the loop worked. The check looks at a single entry from the updated test set and confirms that four keys are present together: the original instruction, the input, the reference output, and a model_response field whose value is the model's answer. Seeing all four confirms that the response went into the correct dictionary and that the original fields were not disturbed.

The generation loop calls the model once per entry, so the total run time depends heavily on the hardware that executes those calls. The source reports the run time for the 110-entry test set on two machines. The progress bar in the run output counts entries as they finish and reports the average speed in iterations per second.

The same code runs on both machines, so the difference comes from the hardware rather than from any change in the logic. When you plan a test run, the number of entries and the speed of the device together determine how long you will wait.