Decoding Strategies for Controlled Text Generation

This article starts with behavior rather than theory. You run a simple text generation call that continues a short starting phrase, observe what it prints, and then work out the rule that explains the result. The call takes the starting phrase, converts it into a sequence of token IDs, extends those IDs with new tokens, and converts the result back into text for printing. Each stage of that pipeline matters for the questions later in this article.

The source assumes that the model, the configuration GPT_CONFIG_124M, the functions text_to_token_ids and token_ids_to_text, and the tiktoken library already exist from earlier in the course. This article does not redefine them. Treat them as given inputs to a single call. Notice that the starting phrase enters the call as a sequence of token IDs produced by text_to_token_ids, not as raw text.

A large language model generates text one token at a time. At each step, the model produces a set of output scores over its vocabulary, and a rule must turn those scores into one actual choice. The source describes generation as a sequence of steps, one for each new token. The generation call takes a parameter that sets how many new tokens to produce. In the call described in the source, that parameter is max_new_tokens=25.

The step that matters most is what happens after a token is chosen. Each selected token becomes part of the context that informs the next prediction. This is why the output extends the starting phrase instead of being a fixed reply. The loop is: score the vocabulary, select one token, append it to the context, and repeat until the token budget is used up. The source describes this behavior but does not show the internal code of the loop, so the diagram below summarizes behavior, not an implementation.

The source gives the selection rule precisely. At each generation step, the model assigns a probability score to every token in its vocabulary, and the generated token is the one with the largest probability score among all of them. The choice is not a random draw from the distribution. It is always the top-scoring candidate. This single rule decides which word-piece comes next under the default strategy.

This procedure is called greedy decoding. Greedy decoding converts model outputs into probability scores and selects the token with the highest score. The source states that, by default, the next token is generated using greedy decoding. Unless a different strategy is specified, the model therefore takes the top-scoring candidate at every step, and the selection rule contains no randomness.

Because greedy decoding always picks the same token for the same context, repeated runs on the same starting context produce the same outputs. This gives you a debugging rule. If a generation result changes between runs, the cause is in the inputs, not in randomness inside the selection rule. The source names the inputs to check: the starting context, the generation settings, the model, and the tokenizer.

Greedy decoding is not the only option. Probabilistic sampling is a different decoding strategy that works from the same probability scores but does not fix the choice to the single highest one. The source gives its purpose: using probabilistic sampling, along with temperature scaling, we can influence the diversity and coherence of the generated text. The two techniques are presented together because they are applied as a pair when shaping how the model writes.

The choice of strategy matters because two very different outputs can come from the same model simply because the decoding rule differs. The model's scores can be identical while the written text is predictable under greedy decoding and varied under sampling. Knowing which rule is in effect is therefore part of explaining any generated result.

Choosing a decoding strategy raises a practical question: how do you know whether the generated text is good? The source gives one approach. Training and validation set losses can be used to gauge the quality of text generated by an LLM during training. The losses measured on the training and validation data serve as a signal about how well the model is producing text while it is being trained.

The two halves of the topic connect here. Decoding strategies determine how text is produced from the scores, while loss measurements give a way to gauge the quality of the text the model generates. The source presents these as separate pieces of information, so keep the decoding rule and the quality signal distinct when you reason about a result. The source does not describe a specific pattern in the loss curves that indicates good generalization, so this article does not claim one.