Extracting Model Responses

Once a pretrained model is loaded, the next question is whether it can actually answer an instruction. In this step, the model is given the formatted input for one test set entry, asked to continue it, and the continuation is read back as text. That text is then placed next to the expected answer from the test set so the two can be compared.

The method is the same generate function that was used earlier to pretrain the model, so the familiar conversions between text and token IDs appear again. Three names carry the text forward through the program: input_text, token_ids, and generated_text. This article follows one pass through that loop closely, because the most useful skill here is knowing exactly which line produced which piece of text. When a response looks wrong, the cause is almost always found by tracing where the text changed shape.

The source loop handles the first three entries of the test data. For each entry it formats the input, generates token IDs, converts those IDs back to text, and cleans up the response. Notice that the response is produced by slicing the generated text, not by the model returning a separate answer string. The model never receives the raw string directly: the formatted input is first converted into token IDs, generate produces a longer list of token IDs, and only then is that list converted back into text.

The generate function sits between the two conversions. It takes the encoded input, along with the model and the length limits, and returns token IDs. Decoding happens afterward, so the text you finally read was built from the full token sequence rather than from the model's answer alone. Keep the sequence in mind: text, then token IDs, then generated token IDs, then text again, then the slice, then the cleanup.

The generated text contains both the input and the model's continuation. This is expected behavior, not a bug. A pretrained model is built to continue text coherently, and for a text-completion model the most natural output is a continuation that reads smoothly from the prompt. Returning the prompt together with the continuation therefore produces legible text.

Follow one test entry from start to finish, keeping a short written record of each variable. A useful habit from the source is to note the input text, the token IDs, the generated text, and the response at every line. The first record that looks unexpected tells you which line to inspect.

The most important line in the loop is the slice generated_text[len(input_text):]. The source code assumes the generated text starts with the formatted input, which is why the code can measure the input's length and cut it off. The slice subtracts the length of the input from the start of generated_text, so it skips the first len(input_text) characters and keeps the rest.

The two operations happen in a fixed order. The slice runs first, and strip() acts on what the slice returned. The slice does not change the prompt inside generated_text. Slicing produces a new string, so generated_text still contains the prompt after the slice, and only response_text holds the isolated answer.

The strip() function removes leading or trailing whitespace from the sliced result. It does not remove the prompt itself. That is why the slice must come first: if strip() were applied to the full generated text, the prompt would still be there, and the length-based cut would no longer line up with the original input.

The source also removes the string "### Response:" during cleanup. That marker is part of the prompt format rather than the answer, so it is removed along with the surrounding whitespace before the value is printed.

Response generation is compared against the expected test set answers, so a mismatch points to the stage where the text changed unexpectedly. The fastest diagnosis is to check each value in the pipeline in order, and the first stage whose value differs from what you expected is where the result diverged.