Implementing Transformer Blocks in GPT Architecture

This article is about running a TransformerBlock on sample data and checking what comes back. The approach is to look at the code first, predict what it should produce, and then confirm the prediction from the real output. Each claim here can be checked by running the same few lines yourself.

The sample input is built with the line x = torch.rand(2, 4, 768). Its shape is described as [batch_size, num_tokens, emb_dim]. Read in that order, the three numbers mean: 2 sequences in the batch, 4 tokens in each sequence, and 768 values for each token. Each dimension answers a different question about the data.

The last dimension is the embedding dimension. This is why the source uses 768 as the embedding size for this configuration. Each of the four token positions in each sequence is represented by one vector of 768 values. If you change only one number, you change only one property of the data: the batch size changes how many sequences you process, the token count changes how long each sequence is, and the embedding size changes how many values describe each token.

The snippet is short. It creates a random input tensor, passes it through a TransformerBlock, and prints the shape of both the input and the output. The source gives the configuration as GPT_CONFIG_124M and assumes that both the TransformerBlock class and this configuration already exist. This step only uses them.

The random values in x will differ between runs, but the printed shapes are the thing to check. The source reports both shapes as torch.Size([2, 4, 768]), so the printed output confirms that the block keeps the input shape. The source describes the block only as a component that processes input data, so this step does not make claims about what happens inside it.

A shape check is a comparison, not a guess. Printing both shapes side by side gives two values you can compare position by position: the first number with the first, the second with the second, and the third with the third. When every position matches, the block has preserved the shape.

In the source's run, the two boxes carry identical shapes. The batch size is still 2, the token count is still 4, and the embedding dimension is still 768. The diagram shows only the single call and its two shapes; it does not claim anything about the steps between them.

When a shape check does not match what you expect, the useful question is where in the code the result diverged. In this snippet there are four stages: the input is created, the block is constructed, the block is called, and the shapes are printed. The printed lines are the last stage, so a mismatch that shows up between the input and output lines points to the call output = block(x) or to the block itself, not to the print statements.

The pipeline shows the order of the four stages. The shape check is most informative at the last stage, where both shapes are printed. Use the printed values to narrow the search before you change any code. If the two lines differ, the call output = block(x) is the first place to look, because that single line is where the block receives x and returns output.