Neural Machine Translation of Rare Words with Subword Units

Language models do not read text as whole sentences. They read a sequence of tokens, and the tokenizer decides where one token ends and the next begins. That decision matters more than it first appears. It shapes how much text fits into a model's input, how common and rare words are represented, and how the model handles spellings it has never seen before.

Consider the two obvious designs. If a tokenizer treats every whole word as one token, it needs a very large vocabulary, and any word it did not see during training has no entry at all. If it treats every character as one token, the vocabulary stays small, but sequences become long and individual characters carry little meaning on their own. Subword units sit between these two extremes. Common words can stay whole, while rare or unfamiliar words are assembled from smaller pieces that the vocabulary already contains.

Byte Pair Encoding, usually abbreviated BPE, is one of the tokenization methods used in natural language processing. It is the method behind the tokenizer used to train GPT-2. The name points to its core operation: the tokenizer works on pairs of adjacent symbols and combines frequent pairs into single units. Because the units it builds are pieces of words, BPE belongs to the family of subword tokenization methods.

The method was introduced to machine translation in the 2015 research paper titled Neural Machine Translation of Rare Words with Subword Units by Sennrich et al., available on arXiv. The title states the motivation directly: rare words are the hard case, and subword units are the proposed way to handle them.

BPE starts from a basic set of symbols and looks for the adjacent pair that appears most often. Before reading on, try this on a tiny sequence. The symbols below are a placeholder alphabet, not real text, so you can focus only on the counting rule.

The guess you just made is exactly the decision BPE repeats at every step. The method does not look at meaning. It only counts adjacent pairs, picks the most frequent one, and records the result as a new unit.

A BPE tokenizer is easiest to understand as a pipeline with two stages: training and application. During training, the method starts from a basic set of symbols and repeatedly merges the most frequent adjacent pair into a new symbol. Each merge adds one entry to the vocabulary. The merge itself is a simple replacement: every occurrence of the chosen pair in the current sequence is replaced by one unit.

Notice what the merge did not do. It did not decide what ab means. It only recorded that the adjacent pair occurred often enough to be worth a single unit. The meaning of a token emerges from the sequence statistics of the training text, not from any rule about the symbols themselves.

The loop between counting and merging is the heart of the method. Each pass looks at the current sequence of units, finds the pair that occurs most often, and replaces that pair with a single unit. Because the merged unit becomes a building block for later merges, longer units can emerge from shorter ones over many passes. The example here is deliberately small. Real training runs over a large body of text and performs many merges, but the rule at each step is the same.

Round 2 uses the sequence produced by Round 1, ab ab ab c. Counting its adjacent pairs gives ab with ab occurring twice and ab with c occurring once. The pair ab ab is therefore the most frequent, and merging it produces the unit abab. Scanning left to right and merging non-overlapping occurrences turns ab ab ab c into abab ab c. The vocabulary now holds a, b, c, ab, and abab, so a unit that is longer than one character has been built from a unit that was itself built from two characters.

The key-value view makes the decision rule explicit. Each adjacent pair is a key, and its count is the value. The merge chooses the key with the largest value. Ties are not discussed in the source material, so this article does not assume a tie-breaking rule.

Training and application are distinct stages. Training builds the vocabulary by running the counting and merging loop over a body of text. Application uses the finished vocabulary to split new text into tokens. Once training is complete, the merge list is fixed, and new input is cut using the units that training produced.

This is how a rare word is handled. A word that was never seen during training does not need its own entry. It is assembled from smaller pieces that the vocabulary already contains. The source describes this as combining smaller pieces the vocabulary holds, rather than requiring a dedicated token for every possible word.

The source material points to three concrete resources. Each one lets you move from the concept to real code or to a visual demonstration, and each suits a different goal. Use the concept in this article as the mental model, then turn to those resources to see the same counting and merging loop run on real text.