Model evaluation during training
The number says how the model is scoring.
Why Training Numbers Leave a Gap
Training a language model produces a stream of numbers. Those numbers are useful, because they tell you whether training is making progress and let you compare one stage of training with another. But they do not show what the model is actually writing. A number can improve while the text the model produces still reads poorly, and a number alone gives you no way to see that.
This article introduces a helper function, generate_and_print_sample, that closes that gap. It takes a short piece of starting text, asks the model to continue it, and prints the continuation so that a person can read it. Your job is to keep track of what the function does to the model, to the input, and to the output at each step.
What do you think happens?
While the model writes its continuation inside generate_and_print_sample, which mode should it be in: training mode or evaluation mode?
Reveal answer
Evaluation mode
The function switches the model to evaluation mode before generating, and only switches it back to training mode before returning. The generation step therefore runs in evaluation mode.
Two Signals That Answer Different Questions
The function is designed to sit beside evaluate_model rather than replace it. The evaluation helper produces a numeric estimate of training progress. That number is convenient for comparing one training stage with another, but it does not reveal what kind of text the model is producing. The generated sample fills that gap by giving a concrete piece of model output that a person can read and judge.
Used together, the two signals answer different questions. The number says how the model is scoring. The sample says what the model is writing. Neither one is complete on its own, and the comparison below shows where each one is strong.
| Signal | Produced by | Question it answers | Form of the result |
|---|---|---|---|
| Numeric estimate | evaluate_model | How is the model scoring during this stage of training? | A number that can be compared across stages |
| Generated text sample | generate_and_print_sample | What is the model actually writing right now? | Readable text that a person can judge |
The numeric estimate and the generated sample reveal different things about the model during training.
Tracing the Path From Text to Printed Sample
The function is short, but it has four stages that matter for understanding its behavior. Each stage hands its result to the next, so an error in one stage shows up as a strange result later. The stages are preparing the input, generating new tokens, turning tokens back into text, and printing. The diagram below shows that order, and the subsections that follow walk through each stage in the order the code runs it.
Watching the Model Switch Modes
The function does not leave the model in whatever state it happens to be in. It switches the model to evaluation mode before generating, and it switches the model back to training mode before returning. This is what preserves the surrounding training process: the sample is produced without leaving the model in a different state for the training loop that continues afterward.
A Worked Trace of One Call
Tracing a single call to generate_and_print_sample
A training run calls generate_and_print_sample with a short starting text. Trace what happens, in order, from the moment the call starts to the moment the printed sample appears and the function returns.
- 1Start in training mode
The call begins while the surrounding training process is running, so the model is in training mode when the function is entered.
- 2Encode the starting text
The starting text is converted into token IDs. This is the preparation stage, and the generation step cannot start until these IDs exist.
- 3Switch to evaluation mode
Before generating, the model is switched to evaluation mode.
- 4Generate up to 50 new tokens
The model produces new tokens one at a time, appending each to the context before producing the next. The process covers at most 50 new tokens.
- 5Decode the new tokens
The generated token IDs are turned back into text.
- 6Print the continuation
The decoded text is printed so that a person can read it and judge what the model is writing.
- 7Switch back to training mode and return
Before returning, the model is switched back to training mode, so the training process continues as it was.
The model ends in training mode, the training process continues unchanged, and a readable continuation of up to 50 new tokens has been printed.
Locating Where a Bad Sample Came From
When a generated sample is not what you expected, the useful question is which stage produced the surprise. Because each stage hands its result to the next, you can trace the problem backward through the pipeline. The points below pair each kind of symptom with the stage it most likely points to. These pairings are a reasoning aid drawn from the four-stage structure, not a complete list of every possible failure.
Common mistake Assuming the printed text is the root cause without checking earlier stages
What it looks likeA sample prints as strange text, and the first guess is that the model is broken.
Why it failsThe printed text is the last stage. Any problem that started in encoding, generation, or decoding will appear there, so the print statement itself is rarely where the fault begins.
Do this insteadTrace backward through the stages in order: printing, decoding, generation, then preparing the input.
Common mistake Treating the sample as a replacement for the numeric estimate
What it looks likeJudging the training stage only by reading one generated sample.
Why it failsA single sample shows what the model writes in one case, while the numeric estimate from
evaluate_modelis designed for comparing stages. Each answers a different question.Do this insteadRead the numeric estimate and the sample together, and use each for what it reveals.
Common mistake Assuming the model is in the wrong mode after the call
What it looks likeBelieving that the call leaves the model in evaluation mode for the rest of training.
Why it failsThe function switches the model back to training mode before it returns.
Do this insteadExpect the model to be in training mode after the call returns, and check the mode switches if the training loop behaves unexpectedly.
Diagnosing a Garbled Mid-Training Sample
During a training run, the numeric estimate from evaluate_model looks steady, but a sample printed by generate_and_print_sample reads as garbled text even though the starting text was ordinary. Using the four stages, decide which stage is the most likely origin of the problem, and explain how you would confirm it by tracing the data backward.
Hints
- Ask whether the starting text was converted into token IDs correctly, or whether the problem appears only after the IDs were produced.
- Consider the stage that turns token IDs back into text. A mismatch there would show up in the printed output even if the generated IDs were fine.
- The numeric estimate being steady tells you something about how the model is scoring, but it does not by itself show what the model is writing.
Putting the Pieces Together
- 1
generate_and_print_sampleencodes a starting text into token IDs, generates up to 50 new tokens, decodes them back into text, and prints the result. - 2The model is switched to evaluation mode before generating and back to training mode before returning, so the surrounding training process is preserved.
- 3The printed sample complements the numeric estimate from
evaluate_model: the number shows how the model is scoring, and the sample shows what it is writing. - 4When a sample looks wrong, trace the data backward through decoding, generation, and encoding to find the stage where the result diverged.
Key Takeaways
generate_and_print_sampleencodes a starting text into token IDs, generates up to 50 new tokens, decodes them back into text, and prints the result.- The model is switched to evaluation mode before generating and back to training mode before returning, so the surrounding training process is preserved.
- The printed sample complements the numeric estimate from
evaluate_model: the number shows how the model is scoring, and the sample shows what it is writing. - When a sample looks wrong, trace the data backward through decoding, generation, and encoding to find the stage where the result diverged.