Text Generation using GPT Models

This article follows one short text generation run from start to finish. Rather than starting with definitions, we begin with the result: a GPT model is given a piece of text and asked to continue it. The output of that run is a list of integers, not readable words. The goal is to understand what the model returned and then work backward through each part of the run to see how that result was produced.

Think of the run as a pipeline with three visible stages: prepare the model, call the generation function, and read the output. Each stage can go wrong in a different way, and the output length gives you a way to tell which stage is responsible when something looks off. We will return to that idea later in the article.

Before generating anything, the model is placed into evaluation mode by calling model.eval(). The source describes this call as the switch that disables random components such as dropout. In other words, calling model.eval() changes how the model behaves for the rest of the run.

Dropout is a training-time mechanism. During learning, it introduces randomness so the model does not depend too heavily on any single internal component. When the model is only being run to produce output, that randomness is not wanted, because the same input should not give a different continuation simply because a random component was switched on. Evaluation mode turns that randomness off.

The generation step is a single call to generate_text_simple. It takes four pieces of information: the model to use, an encoded input tensor named idx, a limit on how many new tokens to add through max_new_tokens, and a context size taken from the model configuration as its context length. Each argument answers a separate question. Which model should produce the continuation? What are the starting tokens? How far should the model continue? How much of the preceding text may the model look at?

The function returns output token IDs. It does not return readable words. To read the result as text, the IDs must be mapped back through the tokenizer's vocabulary. Keep this distinction in mind: the printed output is a sequence of integers, and seeing integers is the expected result at this stage of the run.

Generation works by growing the token sequence. The input begins with its own tokens, and on each iteration one new token is added to the end. The number of iterations is capped by max_new_tokens. The diagram below traces the length of the sequence through the run, using the values from the source: an input of 4 tokens and a limit of 6 new tokens.

Each iteration adds exactly one token, so the sequence length rises by one at every step. After six iterations, the sequence has grown from 4 tokens to 10. The earlier tokens are never removed; the model's continuation is simply appended to what was already there. This is why the final output contains the original input followed by the newly generated tokens.

The reported output length of 10 equals the 4 input tokens plus the 6 newly generated tokens. Because this relationship is predictable, the output length is a useful checkpoint. If you know the input length and max_new_tokens, you know what length the output should have. When the printed length differs from that expectation, something upstream has changed.

The useful question is not only what was printed, but at which step the printed value stopped matching what the code should produce. Because the length grows by one per iteration, you can compare the observed length against the expected length at each checkpoint. The first point of divergence narrows down where the problem entered the run.