Subword Tokenization and Vocabulary Design

Language models do not read text as whole sentences. They read a sequence of tokens, and something has to decide where one token ends and the next begins. That something is the tokenizer. Its choices affect how much text fits into a model's input, how common words and rare words are represented, and how the model handles spellings it has never seen before.

Byte Pair Encoding, usually abbreviated BPE, is one of the tokenization methods used in natural language processing. It is the method behind the tokenizer used to train GPT-2. This article builds a mental model of what BPE does and why it is designed the way it is. Once that model is clear, the implementations and interactive demonstrations you meet later will make sense.

There are two simple ways to cut text, and each fails in a different way. If a tokenizer treats every whole word as one token, it needs a huge vocabulary, and any word it has not seen during training has no entry at all. If it treats every character as one token, the vocabulary stays small, but the sequences become long and the individual units carry little meaning on their own.

Subword units sit between these two extremes. Common words can stay whole, while rare or unfamiliar words are assembled from smaller pieces that the vocabulary already contains. The name of the method points to its core operation. A tokenizer built this way works on pairs of adjacent symbols and combines frequent pairs into single units. Repeating that combination gradually builds a vocabulary of units that are longer than one character and shorter than a whole word. Because these units are pieces of words, BPE belongs to the family of subword tokenization methods.

A BPE tokenizer is best understood as a pipeline with two distinct stages. Training builds the vocabulary. Application uses that finished vocabulary to split new text. The two stages are separate: the vocabulary is produced once during training, and afterward it is applied to whatever text needs to be tokenized.

During training, the method starts from a basic set of symbols and repeatedly merges the most frequent adjacent pair into a new symbol. Each merge adds one entry to the vocabulary. During application, new text is cut using the vocabulary that training produced. The source material places BPE in this sequence of stages, with raw text going in and token IDs coming out for the model to read.

The example below is deliberately small. Real training runs over a large body of text and performs many merges, but the rule at each step is the same: find the most frequent adjacent pair, merge it, and record the new unit. Imagine a tiny training text made of the words low, low, and lower. We start with every character as its own symbol.

A single merge is only the first pass. Once lo exists, it can take part in the next count. In the same tiny text, the pair lo followed by w now occurs 3 times, which is more than any other adjacent pair. Merging it produces the unit low, and the word lower becomes low, e, r. A unit that is longer than a single merge could have produced has now emerged from shorter pieces.

BPE was introduced for machine translation. The research paper that introduced this approach is titled Neural Machine Translation of Rare Words with Subword Units. It was published in 2015 by Sennrich et al. and is available on arXiv. The title states the motivation directly: rare words are the hard case, and subword units are the proposed way to handle them.

Today BPE is part of how language models are prepared to read text. It is the method behind the tokenizer used to train GPT-2, which is the concrete industry use the source material names. In that setting, the tokenizer sits between raw text and the model's input IDs.

The source material points to three concrete resources for moving from this concept to real work. Each suits a different goal, ranging from reading the original method to inspecting real code or trying a visual demonstration. The pack does not name those three resources, so check the course's resource list for their current names and links rather than relying on a memory of them.