Building a Real Transformer Block

This lesson starts from running code rather than from a definition. The model you will meet is a class called DummyGPTModel. It is a stripped-down, GPT-style network built with PyTorch. Some of its internal parts are deliberately fake, so the code runs from start to finish even though those parts do no real work yet. Run the listing that accompanies this lesson and observe what comes out. Then use the trace in the following sections to explain every number you saw.

DummyGPTModel is a simplified GPT-style nn.Module. Its constructor builds token and positional embeddings, a dropout layer, a stack of transformer blocks, a final layer normalization, and a linear output head. The forward method is the single place where the order of operations is written down, so reading it from top to bottom gives you the data flow of the whole model.

The model receives a tensor of integer token indices, called in_idx. The stages then run in a fixed sequence. First, token indices are turned into vectors by the token embedding table and by the positional embedding table. Those two results are added together. Next comes the dropout layer, then the stack of transformer blocks, then the final normalization, and finally the linear output head, which produces the logits. Each box in the diagram below is a named component from the class, and each arrow is a value handed from one stage to the next.

Integer token indices are not meaningful to the network as plain numbers, so the model first converts them into vectors in two separate ways. The token embedding table maps each token index to a vector, and this tells the model what each token is. The positional embedding table maps each position in the sequence to a vector, and this tells the model where that token sits. The two tables are indexed differently, one by token identity and one by position, so they answer different questions.

The forward method then adds the two tensors element by element. The result has the same shape as either input, so the next stage receives one combined vector per token position. This sum is taken before any dropout is applied. Nothing else runs between the lookups and the addition, which is why the summed embeddings are the input to every stage that follows.

DummyTransformerBlock and DummyLayerNorm are placeholders. Each one accepts whatever it is given and returns it without modification. Their constructors take the configuration or the normalized shape, so they have the same interface as the real components, which lets the rest of the model be written and run now. But they keep no logic, so no arithmetic happens inside them.

The only stage that transforms the data at the end is the linear output head. It maps each position's embedding vector to a score for every token in the vocabulary, and those scores are called logits. The hidden-state tensor enters with one vector per position and leaves with one score per vocabulary token for each position. In shape terms, the last dimension changes from the embedding width to the vocabulary size, while the batch and sequence dimensions carry through unchanged.

Suppose the final logits look wrong. The method is to compare intermediate tensors stage by stage, in forward-pass order, and to find the first stage where the result diverges from what you expected. Because every stage passes its output to the next, an error introduced early will carry forward, so the earliest deviation is the one to investigate. The steps below follow the order of the forward method.

Read the forward method top to bottom to see the order of stages. Token and positional embeddings are looked up separately and summed before anything else happens. The placeholder transformer blocks and placeholder layer norm return their input unchanged, so only the embeddings, dropout, and the output head shape the logits in this version. When an output is wrong, check the intermediate tensors in forward-pass order and find the first stage where they diverge.