Tokenizing Text Using Byte Pair Encoding

A GPT model cannot read the sentence Every effort moves you the way a person does. Its internal arithmetic operates on numbers, so every piece of text must be converted into numbers before the model sees it. This article follows that conversion for two short input texts, from the moment a string enters the pipeline to the moment the model receives a numerical input it can process.

The journey has two stages that are easy to blur together. The first stage turns text into token IDs, which are integers. The second stage turns those integers into numerical vectors called token embeddings. Keeping these two stages separate in your head is the most useful habit for debugging the pipeline, and the rest of this article is organized around that separation.

The program for this lesson builds a batch from two text inputs. It tokenizes each text with the tiktoken tokenizer, converts each result to a tensor, and stacks the tensors into a single batch. Before reading the explanation, trace the program from its first line to its last and keep track of what each variable holds after each line runs. Pay particular attention to the moment a list of integers becomes a tensor, and the moment two tensors become one.

Each text is tokenized separately, and each resulting list of IDs is wrapped in a tensor. The two tensors are appended to a Python list named batch. The final step, torch.stack with dim set to 0, joins the two one-dimensional tensors into a single two-dimensional tensor. Dimension 0 is the batch dimension, so each text occupies one row of the result. The source output for this batch is a tensor with two rows of four token IDs each, which is the $2 \times 4$ shape. The first row corresponds to the first text and the second row to the second text, so reading the rows in the right order is what lets you connect each output back to its original input.

The call tokenizer.encode(txt) takes a single string and returns a sequence of integers. Each integer is a token ID that stands for one token in the vocabulary of the tokenizer. The output is therefore not a string and not a vector. It is an ordered list of whole numbers, with one number per token, in the same order as the tokens appear in the text.

The second stage takes each integer token ID and selects a numerical vector for it. Token embeddings are typically 768-dimensional for the smallest GPT-2 model, which means each token is represented not by one number but by a vector of 768 numbers. The source figure for this step shows only the first two dimensions of such a vector for readability. The structural point is that the embedding step expands each single integer into a long vector, so the model's input becomes a richer numerical description of each token.

The whole route for a single input runs in a fixed order. Text is tokenized into IDs, the IDs are converted into embeddings, the embeddings are fed into the GPT model, and the model returns output vectors that are postprocessed into output text. In the example the goal is to generate the next word, which is forward.

Notice that the number of input tokens matches the number of output tokens. For this reason, the first token, Every, is not contained in the output. The model produces one output vector for each input position, and the continuation it is asked to generate follows from the sequence it was given. So if you count the tokens going in and find them unchanged coming out, that is expected behavior, not a sign of a fault.

In earlier code the token embedding was handled inside the GPT model itself. In a large language model, the embedded input dimension typically matches the output dimension, and the output embeddings represent the context vectors discussed in chapter 3. This means the model returns one output vector for each input token embedding.