Token Embeddings and Positional Information

A GPT model cannot read a sentence the way a person does. Its internal arithmetic operates on numbers, so every piece of text must be converted into numbers before the model sees it. This article follows that conversion for two short input texts, from the moment a string enters the pipeline to the moment the model receives a numerical input it can process.

The journey has two stages that are easy to blur together. The first stage turns text into token IDs, which are integers. The second stage turns those integers into numerical vectors called token embeddings. Keeping these two stages separate in your head is the most useful habit for debugging the pipeline, and the rest of the article is organized around that separation.

The program for this lesson builds a batch from two text inputs. It tokenizes each text with the tiktoken tokenizer, converts each result to a tensor, and stacks the tensors into a single batch. As you read, keep track of what each variable holds after each line runs. The two moments that matter most are the moment a list of integers becomes a tensor, and the moment two tensors become one.

The output of encode is therefore neither a string nor a vector. It is an ordered list of whole numbers, with one number per token, in the same order as the tokens appear in the text. Order matters: the first integer corresponds to the first token of the text, and so on. This is why you can later connect each output back to its original input.

Tensors are the containers the model works with. Once each text has its own list of token IDs, each list is wrapped in a tensor. The two one-dimensional tensors are then appended to a Python list named batch. Finally, torch.stack with dim set to 0 joins them into one two-dimensional tensor. Dimension 0 is the batch dimension, so each text occupies exactly one row of the result.

Token IDs are single integers, but the model needs a richer description of each token. The second stage converts each integer into a token embedding, which is a numerical vector. In the smallest GPT-2 model, token embeddings are typically 768-dimensional. That means each token is represented not by one number but by a vector of 768 numbers.

The embedding step expands each single integer into a long vector. The source figure shows only the first two dimensions of such a vector for readability, but the important structural point is the expansion itself. Each ID is a single number going in and a 768-number vector coming out, so the model's input becomes a richer numerical description of each token. In the diagram above, each integer chooses one vector, which is why a sequence of IDs becomes a sequence of vectors in the same order.

The whole route for a single input runs in a fixed order. Text is tokenized into IDs, the IDs are converted into embeddings, the embeddings are fed into the GPT model, and the model returns output vectors that are postprocessed into output text. In the example, the goal is to generate the next word, which is forward.

The model returns one output vector for each input token embedding, so the number of input tokens matches the number of output tokens. Because of this one-to-one correspondence, the first token, Every, does not appear in the output. The model produces one output vector for each input position, and the continuation it is asked to generate follows from the sequence it was given.

When the final output looks wrong, the pipeline gives you a sequence of checkpoints. Each one can be inspected in order, and the first checkpoint whose contents differ from what you expected is where the problem begins. Because the stages are separate, you do not have to guess whether the fault lies in the text, the IDs, the embeddings, or the model.