Text Generation with a GPT Model

A practical question sits at the center of this article: how do you put a number on how good a model's generated text is? Before any number can be computed, the model has to produce its output. The sequence that runs the model on a batch of inputs, converts its raw outputs into probabilities, picks token IDs, and decodes them back into words is the same sequence that generate_text_simple performs internally. Running that sequence on a small batch first lets us see what the model actually produces before we score it.

The batch used here contains two inputs, 'every effort moves' and 'I really like'. Each input has already been converted to token IDs. Each input has three tokens, so the batch holds six token positions in total. Keep that count of six in mind, because it reappears when the loss is averaged.

Text generation is a five-step procedure. Each step consumes the output of the step before it, so the stages form a pipeline. The diagram below traces the path from input text to output text, and the explanation after it describes what each stage hands to the next one.

The first step maps the input text to token IDs using a vocabulary. The second step has the model produce a probability score for every token in the vocabulary at every input position. The third step locates, for each position, the index with the highest probability. The fourth step collects those highest-probability indices as the predicted token IDs. The fifth step maps the predicted token IDs back into text using an inverse vocabulary that goes from IDs to words.

These five steps are the same whether the vocabulary is tiny or large. The example diagram uses a seven-word vocabulary so that every probability vector fits on one page. The GPT model used in this work has a far larger vocabulary, and that difference changes how the numbers should be read.

The probability tensor has a shape that encodes three things at once. The first dimension is the batch size, which is the number of input examples. The second dimension is the number of tokens in each input. The last dimension matches the vocabulary size, because each input position gets one probability for every possible next token.

Read that shape as two examples, three positions per example, and one probability per vocabulary entry at each position. The last number, 50257, is the vocabulary size. Token IDs for this model range from 0 to 50,256. The seven-word vocabulary in the pipeline diagram is only a teaching simplification, so its numbers should not be carried over directly to the GPT model.

Scale matters for the starting values. When probability is spread evenly over the vocabulary, each entry is about 1 divided by the vocabulary size. For the seven-word example vocabulary that is about 0.14, while for the 50,257-word vocabulary it is about 0.00002. This is why the starting probabilities look so small. A learner who expects an untrained model to show values like those of a seven-word vocabulary will think something is wrong. Tiny values are the normal starting point for a large vocabulary.

The targets are the token IDs the model should produce. They are built from the inputs by shifting every sequence one position forward, so that each input token is paired with the token that follows it. The input 'every effort moves' has the target 'effort moves you'. The input 'I really like' has the target 'really like chocolate'. Each target token is the next word after the corresponding input token.

This shifting is what teaches a language model to predict the next token in a sequence. Without it, the model would be asked to reproduce its input, which is a different task.

For every input position, the probability vector has one entry per vocabulary token. The entry at the index of the target token is the value that matters. The training goal is to raise that entry, so that the correct token becomes more likely relative to the others. A model that already picks the right token but gives it only a modest probability still has room to improve, and the loss measures that room.

Before any training, the model's vectors are essentially random. Each input token produces a vector, each vector index corresponds to a vocabulary word, and training is meant to maximize the value at the index of the target token. Because the vectors start random, the values at the target indices are not special yet.

Indexing the probability tensor with the target IDs gives the untrained probabilities assigned to the correct tokens. For the first input, these are roughly 7.45e-05, 3.11e-05, and 1.16e-05. For the second input, they are roughly 1.03e-05, 5.68e-05, and 4.76e-06. Every value is tiny, which matches the expectation for a large vocabulary. These six numbers are the inputs to the loss.

The loss is computed in three ordered steps that start from the six target probabilities. First, a logarithm is applied to each probability. Logarithms make the values easier to optimize than the raw probabilities. Second, the six log probabilities are averaged into a single number. Third, that average is multiplied by -1. The result is the cross entropy loss, which is the quantity training reduces.

PyTorch provides a built-in cross entropy function that performs all of these steps at once, so you rarely need to compute them by hand in practice. Doing the steps manually is still valuable because it shows exactly which numbers feed into the loss and why a larger probability on the correct token lowers it. A larger target probability gives a log value closer to zero, which pulls the average toward zero and the negated loss downward.

When generated text is compared with its target, the first useful step is to decode both sides and read them position by position. Decoding the first batch's targets gives ' effort moves you'. Decoding the model's predicted token IDs for the same batch gives ' Armed heNetflix'. The two strings differ throughout, which is consistent with an untrained model.

A divergence report should identify the first position where the predicted token differs from the target, because that is where the sequence departs from what was expected. The loss value for that position is the place to investigate. In this batch, every decoded output differs from its target, so the investigation starts at the first position of each sequence. A trained model would be expected to match the target at more positions, and the first mismatch would move further along the sequence.

The loss sits between producing text and training the model. Text generation is the first of several stages in the chapter, and the text evaluation stage, where the loss is implemented, comes next. After that come the training and validation stage, the training function, and later stages on text generation strategies and weight handling. The loss serves as the measuring tool that the training function will rely on.