Tokenizing Text for a GPT Model

This article starts from running code rather than from a definition. The model you will meet is a class called DummyGPTModel. It is a stripped-down, GPT-style network built with PyTorch. Some of its internal parts are deliberately fake, so the code runs from start to finish even though those parts do no real work yet. The plan for this lesson is to run the accompanying listing, observe what comes out, and then use a step-by-step trace to explain every value you saw.

The source material describes the model's behavior in prose rather than reproducing its full listing, so this article explains the structure and data flow that the class is built around. When you run the listing yourself, use the trace in the sections below to account for each stage. If an output looks wrong, the same trace tells you where to look.

The model receives a tensor of integer token indices, called in_idx. Those indices are not meaningful to the network as plain integers, so the model first converts them into vectors. After that conversion, the data passes through a fixed sequence of stages: the embeddings, a dropout layer, a stack of transformer blocks, a final normalization, and the output head. The forward method is the single place where this order is written down, so reading it from top to bottom gives you the data flow.

Read the diagram as a chain of hand-offs. Each box is a named component of the class, and each arrow is the value passed from one stage to the next. The stage that matters most for debugging is the first one, because every later stage depends on the vectors it produces. If the embeddings are wrong, nothing downstream can fix them.

The model turns each integer index into a vector in two separate ways and then adds the results together. A token embedding maps each token index to a vector, which tells the model what the token is. A positional embedding maps each position in the sequence to a vector, which tells the model where that token sits. Because the two tables are indexed differently, one answers the question what and the other answers the question where.

The forward method adds the two tensors element by element. The result has the same shape as either input, so the next stage receives one combined vector per token position. This is why the sum can be passed directly onward without any reshaping. The source also states that the sum is taken before any dropout is applied, so dropout operates on the combined vector and not on either embedding alone.

After the embeddings are summed, the data moves through four more stages. Two of them are placeholders. The class DummyTransformerBlock and the class DummyLayerNorm each accept whatever they are given and return it without modification. Their constructors accept a configuration or a normalized shape, so they match the interface of the real components. They keep no logic, so the value that goes in is the value that comes out.

The transformer blocks are gathered into an nn.Sequential container. The container is built from a list comprehension that creates n_layers copies of the placeholder, and each copy receives the output of the previous one. Because every copy is an identity operation, the whole stack is also an identity operation. Adding more layers does not change the data, because passing a value through an identity any number of times still returns that value. The final normalization is a single placeholder, so it adds nothing either.

The only stage that transforms the data at the end is the linear output head. It maps each position's embedding vector to a score for every token in the vocabulary, and those scores are called logits. In this placeholder version, the logits therefore come directly from the summed embeddings and the output head. The placeholders in between have no effect, which is what makes the trace below manageable.

Notice also what this implies about where to look. If the logits look surprising, the placeholders cannot be the cause in this version, because they pass values through untouched. The suspects are the stages that actually compute something: the embedding lookups, the sum, and the output head.

DummyGPTModel is a single nn.Module, but it contains several sub-components. Its construction builds the token and positional embeddings, a dropout layer, a stack of transformer blocks, a final layer normalization, and a linear output head. The forward method then calls these parts in order. Seeing this as a containment structure helps you know which part owns which values when you inspect the model.

When the output is not what you expected, the source's advice is to compare intermediate tensors stage by stage in forward-pass order. You are looking for the first stage where the result diverges from what you expect. Earlier stages are checked first, because a divergence in an early stage carries forward into every later one. This turns a vague problem, the output looks wrong, into a short sequence of concrete checks.