Tokenization and Token IDs

A text generation call takes a short starting phrase and returns that phrase extended with new text. Before learning the rule that controls this behavior, it is worth making a prediction, because the answer tells you what the rest of the article needs to explain. Imagine running the same generation call three times in a row, with the same model, the same starting phrase, and the same settings.

The rest of this article explains why that answer holds. It also shows which inputs you should check when a result looks different from what you expected. The model is never run in isolation here: the starting phrase goes in, a generation function extends it, and the result is converted back to text for printing.

The starting phrase is not passed to the generation function as raw text. It is first converted into a sequence of token IDs. A token is a word-piece from the model's vocabulary, and each token has an integer ID that identifies it. The conversion from text to token IDs is handled by a helper named text_to_token_ids, and the reverse conversion from token IDs back to text is handled by token_ids_to_text.

The diagram is illustrative: it shows the shape of the mapping, with three tokens standing in for however many a real phrase produces. The key idea is that the model works with integer IDs. Each ID points into the vocabulary, which is the full set of tokens the model can produce. The generation function extends the sequence of IDs, and only at the end are those IDs turned back into readable text.

Generation is a sequence of steps, one for each new token. The generation call takes a parameter that sets how many new tokens to produce. In the call described in the source, that parameter is max_new_tokens=25, so the loop runs until 25 new tokens have been added. Each step follows the same cycle, and the cycle is what turns a single prediction into a continuation of the starting phrase.

Follow one pass through the loop. The model reads the current context, which is the starting IDs plus any tokens already generated. It assigns a probability score to every token in its vocabulary. One token is selected and appended to the context. That longer context then feeds the next prediction. Because each new token is added to the context, later predictions are based on everything that came before, which is why the output extends the starting phrase rather than being a fixed reply.

The source states the selection rule precisely. At each generation step, the model assigns a probability score to every token in its vocabulary. The generated token is the one with the largest probability score among all of them. The choice is not a draw from the distribution, where a lower-scoring token might sometimes win. It is always the top-scoring candidate, and this single rule decides which word-piece comes next.

Determinism follows directly from the selection rule. If each step always picks the top-scoring token, and each step starts from the same context, then each step gives the same choice. Step after step, the same tokens get appended, so the full output is the same. This is why repeated runs of the same generation call on the same starting phrase produce the same outputs.

Two runs are identical only when their inputs are identical. The source names the inputs that determine the output: the starting context, the generation settings, the model, and the tokenizer. If any one of these changes, the path through the loop can change, and the printed text can change with it. The same-output guarantee applies to a fixed set of inputs, not to every call that looks similar.

When the generated text is not what you expected, the way to find the cause is to follow the data through the pipeline, one stage at a time. The pipeline starts with the phrase you typed and ends with the printed text. Each stage produces a value, and an unexpected token must have come from some stage that produced a different value than you assumed.