Input Embeddings

A language model does not read a sentence as a single lump. It receives a sequence of tokens, and each token is turned into a vector. Those token vectors describe what each token means. What they do not, by themselves, describe is where each token sits in the sequence. The word at the start of a sentence and the same word at the end would look identical to the token vector alone. Input embeddings are the piece of the pipeline that fills this gap.

This article follows the code first. You will build a small set of positional embeddings, read the shape of the result, and then see how that result combines with token embeddings. The goal is not to memorize a recipe but to be able to trace each step and say what the numbers in the shape mean.

Positional embeddings are one half of the input representation. The other half is the set of token embeddings, which carry the semantic meaning of each token. Think of the two as answering different questions. Token embeddings describe meaning. Positional embeddings describe location. Input embeddings are the combined result that the rest of the model consumes.

The positional embedding layer is built with torch.nn.Embedding, given two numbers. The first is context_length, which is how many positions the model handles. The second is output_dim, which is the size of each vector. The layer behaves like a table with one row per position. Each row is a learned vector of length output_dim.

The positions themselves are supplied by torch.arange(context_length). This produces the sequence of position indices 0 through context_length minus 1. Passing those indices into the layer returns the row for each position. So position 0 looks up row 0, position 1 looks up row 1, and so on. Each position gets its own vector.

Reading the diagram as a lookup table helps. The index 0 through 3 is the key, and the row is the value. Nothing in the table is a token. It is purely a record of position, and each row is learned as the model is trained.

Because each position gets its own vector, the shape of the positional embedding tensor reflects both quantities that define it. The number of rows matches the context length, and the number of columns matches the output dimension. In the source example there are four positions, and each is represented by a 256-dimensional vector. That is why the printed shape describes four vectors of length 256.

The two embedding types meet at a single addition. Token embeddings and positional embeddings are added together, and the sum is called the input embeddings. The input embeddings are the embedded input examples that can be processed by the main LLM modules. The diagram below traces the path from raw text to this meeting point.

The addition works only when the two tensors line up. The shape of positional embeddings should match the shape of token embeddings. In the source example, the token embeddings are arranged as eight batches, each holding a 4 by 256 tensor, so the full token tensor has shape 8 by 4 by 256. The positional embedding tensor is 4 by 256. PyTorch adds that single 4 by 256 tensor to each of the eight batch entries, so every batch receives the same positional information.

The input embeddings keep the token embedding shape. In the source example that shape is 8 by 4 by 256. The addition changes the values inside each vector, but it does not change the overall shape of the result.

The creation steps run in a fixed order: token IDs, then token embeddings, then positional embeddings, then the addition. A mismatch in context length or output dimension shows up at the step where the two tensors first have to agree. For positional information, that is the addition step, because the positional tensor is built from context_length and output_dim and must match the token tensor's shape to be combined with it.