Byte Pair Encoding (BPE) Tokenizer

A large language model cannot read a sentence the way you do. Before any learning can happen, text has to be turned into a form that a deep learning model can process. That preparation happens in two stages. The first stage converts raw text into discrete integer identifiers. The second stage converts those identifiers into continuous vectors that a model can work with. As you read, keep one question in mind: what does the text look like after each step?

The first step is tokenization. Raw text is broken into tokens, and a token can be a word or a character. Tokenization is the general process of breaking text into these units. After that, each token is converted into an integer representation called a token ID. So the pipeline runs from a string of characters, to a list of tokens, to a list of integers.

Notice that the output of tokenization is discrete. Each token ID is a whole number that names one token. The text is no longer a continuous stream of characters but a sequence of labels drawn from a fixed set. That discreteness is what makes the next stage necessary, because whole numbers on their own are not a rich representation for a deep learning model.

Token IDs are integers, and integers alone are not a rich representation for a deep learning model. An embedding layer in PyTorch solves this by functioning as a lookup operation. It takes a token ID and retrieves the vector that corresponds to it. The retrieved embedding vectors provide continuous representations of tokens, and the source describes this as crucial for training deep learning models such as LLMs.

The contrast between the two stages is the key idea. Token IDs are discrete whole numbers, one per token. Embedding vectors are continuous representations retrieved by those IDs. The lookup is what moves a token from a label into a form suited to deep learning.

Tokenization is the general process of breaking text into tokens. BPE is one specific tokenizer that performs that process, and its distinguishing feature is how it treats words that lie outside its vocabulary. A tokenizer built only on whole words runs into trouble when it meets a word it has never seen. BPE handles such words efficiently by breaking the unfamiliar word into subword units or, if necessary, into individual characters, so the word does not have to be discarded.

The name points to the core operation. A BPE tokenizer works on pairs of adjacent symbols and combines frequent pairs into single units. Repeating that combination step gradually builds a vocabulary of units that can be longer than one character and shorter than a whole word. Because those units are pieces of words, BPE belongs to the family of subword tokenization methods.

The approach was introduced to machine translation in the research paper titled Neural Machine Translation of Rare Words with Subword Units, published in 2015 by Sennrich et al. and available on arXiv. Its title states the motivation directly: rare words are the hard case, and subword units are the proposed way to handle them.

There are two obvious extremes. If a tokenizer treats every whole word as one token, it needs a huge vocabulary, and any word it has not seen during training has no entry at all. If it treats every character as one token, the vocabulary stays small, but the sequences become long and the individual units carry little meaning on their own. Subword units sit between these extremes. Common words can stay whole, while rare or unfamiliar words are assembled from smaller pieces that the vocabulary already contains.

This is also why subword units let a tokenizer represent rare words. A rare word does not need its own dedicated entry, because it can be spelled out as a combination of smaller pieces that the vocabulary already holds.

BPE builds subword units by repeatedly merging the most frequent adjacent pair of symbols into a new unit. Each merge adds one entry to the vocabulary. Here is a deliberately small example so you can watch the rule work. Imagine the training text has been split into single symbols, and the sequence is: t h e _ t h e _ t h i s. The underscore marks a space. Scan the sequence for adjacent pairs. The pair t h appears three times, the pair h e appears twice, and the other pairs appear at most once. The most frequent pair is t h.

Notice what the merge did and did not do. It did not look at whole words; it looked only at which adjacent symbols sat next to each other most often. The new unit th is now available as a building block, so later merges can combine it with other units.

The loop between counting and merging is the heart of the method. Each pass looks at the current sequence of units, finds the pair that occurs most often, and replaces that pair with a single unit. Because each merged unit becomes a building block for later merges, longer units can emerge from shorter ones over many passes. In the tiny example above, th could later join with e or with another unit, depending on what the counts show at that point.

The example is deliberately small. Real training runs over a large body of text and performs many merges, but the rule at each step is the same: find the most frequent adjacent pair, merge it, and record the new unit. The material used for this article describes the repeated loop but does not specify a fixed stopping rule, so treat the number of merges as a design decision of the training run rather than something this article fixes.

Suppose the tokenizer meets a word that is not in its learned vocabulary. BPE does not give up on it. The unfamiliar word is broken down into subword units that the vocabulary does contain. If the word cannot be covered by larger pieces, the tokenizer falls back to individual characters. In either case the pieces are still represented, so the word is encoded rather than discarded.

This is why the choice of tokenizer affects how a model handles spellings it has never seen before. A model that receives an unfamiliar word still receives a sequence of tokens it can work with, even if those tokens are smaller than the word itself.

The material for this article points to three concrete resources. Each one lets you move from the concept to real code or to a visual demonstration, and each suits a different goal. Use an implementation when you want to see how the merge loop is written, and use an interactive demonstration when you want to watch merges happen on a text you choose. The original paper by Sennrich et al. is the primary reference for the approach and its motivation for rare words.