Generating Text with GPT

A GPT-style model does not read words the way a person does. It operates on numbers. That means every prompt you give it must first be converted into a sequence of integer token IDs before the model can work with it, and every sequence of token IDs the model produces must be converted back into text before a person can read it. Text in, numbers through the model, text out: this two-way conversion is the subject of this article.

The source focuses on two utility functions that handle those conversions. One turns text into token IDs. The other turns token IDs back into text. Your job is not to memorize the code but to understand what each line is responsible for and to predict what the data looks like at each stage. Before going further, try to predict what happens to a prompt as it moves through the pipeline. Which conversions must happen, and in which order?

The source uses the tiktoken library and requests the gpt2 encoding through tiktoken.get_encoding('gpt2'). The tokenizer object is the component that knows how to map pieces of text to integer IDs and back again. Both utility functions receive the tokenizer as an argument. This matters because the same tokenizer must be used in both directions. If the encoder and decoder disagreed about what each integer means, the text that came out would not match the text that went in.

Think of the tokenizer's vocabulary as a lookup table that runs in both directions. Encoding looks up each piece of text and returns its integer. Decoding takes the integer and returns the text it stands for. Because the table is shared, a round trip through the same tokenizer should recover the original text.

The source code does not show a torch import, and it refers to model and GPT_CONFIG_124M without defining them. Those names come from earlier material in the same chapter and must already exist when the code runs. Treat the snippet above as the two functions in standard Python form, with the calls exactly as the source gives them.

Encoding produces a flat, one-dimensional list of IDs. A flat list of three IDs has a shape of three elements, with no leading axis. The model, however, expects a batch: a leading dimension that holds one or more input sequences. unsqueeze(0) inserts a new axis at position 0. The result has a leading dimension of size 1, which holds the single prompt, followed by the axis of token IDs. The original sequence is unchanged; it is just wrapped one level deeper.

The full round trip follows a fixed order. The prompt starts as a string. The tokenizer encodes it into a list of integer token IDs. text_to_token_ids wraps that list in a tensor and adds a batch dimension so the model receives input with the expected shape. The model call generate_text_simple then produces more token IDs. token_ids_to_text removes the batch dimension, converts the tensor back to a plain list, and asks the tokenizer to decode that list into text. The diagram below shows the sequence and where the model sits within it.

Notice that the model sits in the middle of the chain, between two conversions. It never receives raw text and it never returns raw text. Everything it reads and writes is a sequence of token IDs. This is why the encoding and decoding stages are essential rather than cosmetic.

The two batch-dimension operations are paired. unsqueeze(0) is applied on the way into the model, and squeeze(0) is applied on the way out, before decoding. Because they mirror each other, the decoder receives a flat sequence of IDs rather than a nested one. The tokenizer's decode method expects that flat list, so the squeeze step is what makes the final conversion possible.

Follow the data in that example. The model output still has the batch axis from the input. squeeze(0) drops that axis, flat.tolist() turns the tensor into a Python list of four integers, and tokenizer.decode converts that list into the text those four IDs represent.

This distinction is the core of debugging the round trip. Incoherent output does not automatically mean the conversions failed. The source treats the round trip as something to check separately from the quality of the model's output.

When a generated result looks wrong, check the stages of the round trip in order rather than guessing. The flowchart below suggests a sequence of checks. Each check narrows the cause before you move on, so that a problem in one stage is not blamed on another.

The source raises a natural follow-up question. If incoherent output is expected from an untrained model, how do we define what makes text coherent or high quality, and how do we measure it? The source answers by introducing a numerical method, a loss metric, that evaluates the generated content. The loss metric is described as a progress and success indicator during training, and it is the subject of the next subsection of the chapter. Later chapters will also review additional methods for assessing model quality when the LLM is fine-tuned.