The generate Function in GPT-Style Models
Top-k sampling restricts the model's next-token choice to the top k tokens, and the value of k sets how many candidates remain.
Why the Same Question Gets Different Answers
Ask a language model the same question twice and you may get two different answers. That variation is not a bug in the model. It comes from the way the model chooses each next token. Generation is the process of producing text one token at a time, and the choice made at each step is shaped by a set of generation settings. Top-k sampling is one of the most common of these settings, and it gives you a direct way to control how adventurous the output is.
What do you think happens?
You ask a model the same question twice and get two different answers. What do you think causes the difference: a bug in the model, or something about how each next token is chosen?
Reveal answer
The difference comes from how each next token is chosen. At every step the model has a set of possible continuations, and the generation settings decide how that choice is made. The variation is not a bug.
Because the choice among the allowed candidates is not fixed, sampling normally involves some variation from run to run.
The Ranked Candidate List at Each Step
Think of the model as having a ranked list of candidate next tokens at each step. The strongest candidates sit near the top of the list, and weaker, less likely candidates sit further down. Without any restriction, the model may pick from the whole list, including candidates it considers unlikely.
Consider the prompt 'The cat sat on the' and an illustrative ranked list of next-token candidates. The numbers below are invented for teaching and are not taken from a real model. Trace what the list looks like before any restriction is applied.
| Rank | Candidate token | Illustrative weight |
|---|---|---|
| 1 | mat | 0.40 |
| 2 | rug | 0.25 |
| 3 | sofa | 0.15 |
| 4 | roof | 0.10 |
| 5 | moon | 0.06 |
| 6 | banana | 0.04 |
Illustrative ranked candidates for the next token after 'The cat sat on the' (generated example, values invented for teaching)
What Top-k Sampling Removes
Top-k sampling restricts the model's next-token choice to the top k tokens, and the value of k sets how many candidates remain.
Top-k sampling shortens the ranked list to its strongest entries. The value of k says how many entries survive. Everything below that cut is removed from the candidate pool, so the model can no longer pick those tokens at this step. This restriction is why top-k sampling changes the output: the model's choice is confined to a smaller set of options.
Applying top-k with k set to 3
Using the illustrative ranked list for 'The cat sat on the', which tokens can the model still choose from if k is 3?
- 1Read the ranking
The ranked list is mat, rug, sofa, roof, moon, banana, from strongest to weakest.
- 2Apply the cut
With k set to 3, only the top 3 entries survive the cut.
- 3Identify removed candidates
roof, moon, and banana fall below the cut and are removed from the candidate pool.
- 4State the remaining pool
The model can now choose only among mat, rug, and sofa at this step.
With k set to 3, the candidate pool is mat, rug, and sofa. The tokens roof, moon, and banana can no longer be picked at this step.
How the Size of k Shapes the Output
The value of k controls how much room the model has to vary. The two ends of the range behave quite differently, and the comparison below summarizes the contrast described in the source.
Temperature Working Alongside k
The temperature setting works alongside k. Both shape how the model's choice is made. The source does not present them as one fixed pair for every task. Instead, different tasks are paired with different combinations of k and temperature. For intuition, a temperature below 1 tends to concentrate the probability on the strongest candidates, while a temperature above 1 tends to spread the probability more evenly across the candidates.
Matching Settings to the Task
The right settings depend on what you need the text to do. The source describes two broad goals. The first is precision, where answers must be clear and accurate. The second is imagination, where variety is valuable. The table below summarizes the pairings the source associates with each goal.
| Goal | Example tasks from the source | Top-k setting | Temperature setting |
|---|---|---|---|
| Precision | Formal documents, reports, technical analysis, code generation, question answering, educational content | Low k | Below 1 |
| Imagination | Brainstorming, creative content such as fiction | Larger k, in the range of 20 to 40 | Above 1 |
Pairings of goal, k, and temperature described in the source
Getting Deterministic Output
Sampling normally involves some variation, because the choice among the allowed candidates is not fixed. A deterministic setup removes that variation so that the same input produces the same output on every run. The source states that there are multiple ways to force deterministic behavior with the generate function, and it gives one concrete method.
Why does this remove the variation? Variation comes from picking among candidates. When top-k is disabled and no temperature scaling is applied, the generation process no longer reshapes or trims the candidate choice, so the same input follows the same path on every run.
Mistakes Beginners Make with k and Temperature
Common mistake Treating a small k as the only setting that matters for accuracy
Why it failsThe source pairs low k with temperature below 1 for precision-focused tasks. The two settings work together rather than one replacing the other.
Do this insteadChoose k and temperature as a pair that matches the goal of the output.
Common mistake Using a large k and temperature above 1 for a report or code answer
Why it failsThe source associates larger k with temperature above 1 for brainstorming and creative content, where variety is the aim. Accuracy-focused tasks are matched to low k with temperature below 1.
Do this insteadMatch the settings to the task: precision tasks get low k and temperature below 1; creative tasks get larger k and temperature above 1.
Common mistake Assuming that a careful-looking setting guarantees identical output every time
Why it failsThe source says deterministic output is forced by a specific configuration, top_k=None with no temperature scaling, which is distinct from simply choosing a careful setting.
Do this insteadIf identical output is required, use the explicit deterministic configuration rather than relying on a low k alone.
Common mistake Believing that variation between runs is a fault in the model rather than a result of how each next token is chosen
Trying the Settings on a New Task
A team needs a model to write a short technical analysis that must be accurate and repeatable, and a separate team needs a model to brainstorm story ideas with lots of variety. Using only the pairings in this article, which top-k and temperature settings fit each team, and which setting would you change to make the first team's output identical on every run?
Hints
- Recall which goal, precision or imagination, each task falls under.
- Precision tasks are paired with low k and a temperature below 1; imagination tasks with larger k in the range of 20 to 40 and a temperature above 1.
- For identical output, the source names top_k=None and no temperature scaling.
Key Takeaways
- Top-k sampling restricts the model's next-token choice to the top k tokens, and the value of k sets how many candidates remain in the pool.
- A small k with temperature below 1 suits accuracy-focused tasks such as reports, code generation, question answering, and educational content, while a larger k in the range of 20 to 40 with temperature above 1 suits brainstorming and fiction.
- Deterministic output can be forced by setting top_k=None and applying no temperature scaling, which removes the choice that causes run-to-run variation.
- Variation between runs is not a bug; it comes from the way the model chooses each next token under the active generation settings.