Absolute Positional Embeddings

A tokenizer turns text into integers. Each integer is a token ID that names one entry in a fixed vocabulary. On its own, though, an integer carries no information about how one token relates to another, so a model cannot do much arithmetic with it directly. The usual remedy is to replace each ID with a vector, which is a list of numbers that the model can learn to adjust during training. That replacement is called a token embedding.

In this lesson the token IDs come from the BPE tokenizer built earlier in the course, and the vocabulary has 50,257 entries. The embedding size is set to 256 dimensions. The source describes this as much smaller than the 12,288 dimensions used by the original GPT-3 model, yet still a reasonable size for experimentation on ordinary hardware. The 256 figure is a practical choice for learning, not a claim about what production models use.

Think of the embedding layer as a table with one row for every possible token ID. The vocabulary size sets how many rows the table holds, and the output dimension sets how long each row is. When a token ID arrives, the layer does not compute anything complicated. It selects the row whose position matches that ID and returns the vector stored there. The row is the vector that a given token ID maps to.

The lookup happens for each entry in the input independently. Two tokens with different IDs select different rows, and each selected row is a full 256-value vector. The embedding table is the only place those vectors live, which is why the layer is described by its two dimensions alone.

The constructor call takes the form torch.nn.Embedding(vocab_size, output_dim). In the lesson the first argument is 50,257 and the second is 256. These two numbers fully determine the layer's shape: the weight matrix has $50{,}257 \times 256$ entries, with 50,257 rows and 256 columns. Reading the arguments in order is the first check to make whenever you build one of these layers.

The batch passed to the layer is a two-dimensional grid of token IDs. Each row holds one text sample, and each column holds one token position. The lesson uses a data loader to fetch a batch of 8 samples with 4 tokens each, so the input grid has shape $8 \times 4$. Every entry in that grid is one integer ID between 0 and 50,256.

When you check embedding output, verify the input shape first, then confirm that the final axis equals the output dimension. Tracing the pipeline from the token ID tensor forward shows where a problem first appears. The input shape is set by the data loader, so it is the first place to look. The next place is the embedding layer, whose weight matrix is fixed by the two constructor arguments.

If the vocabulary size is smaller than the largest token ID in the batch, some IDs have no row in the table to select. If the output dimension is set to a value other than 256 in only one of the two embedding layers, the outputs have different final axes and cannot be combined. In both cases the problem appears at the layer whose arguments were set incorrectly, not at the layer after it.

The token embedding layer treats each token by its identity alone. It does not record where in the sequence a token sits. The source notes that a GPT-style absolute embedding approach needs a second embedding layer with the same embedding dimension as the token embedding layer. That second layer supplies position information, and it is the subject of the next step in the lesson.

Both layers must share the 256-dimensional size so that their outputs can be combined. This is the reason the output dimension is a shared design choice rather than a free setting for each layer. The position vector and the token vector for the same slot in the grid are added together, which is why matching final axes matter before any combination happens.

Most errors in this step come from reading the constructor arguments in the wrong order or from assuming the output keeps the input's shape. The items below are the ones most likely to cost time.