Handling Unknown Words in Tokenization

A language model does not read raw text one letter at a time the way a person does. Instead, text is split into units called tokens, and each token is looked up in a fixed vocabulary. The hard design question is what those units should be, and the answer decides what the model can and cannot represent.

Consider the two obvious choices. If every whole word were a unit, the vocabulary would have to list every word that might ever appear, and any word missing from that list would have no representation at all. If every single character were a unit, the vocabulary would be tiny, but every word would become a long sequence of pieces that carries little meaning on its own.

Byte Pair Encoding, usually abbreviated BPE, sits between these two extremes. Its strategy is to build a vocabulary that begins with small pieces and grows by combining the pieces that appear together most often. The result can contain common whole words, useful fragments of longer words, and single characters as a fallback. This article follows that construction one stage at a time, so you can see what the tokenizer is doing rather than memorizing its outcome.

The first stage of BPE is deliberately simple. Before any merging happens, the vocabulary is filled with every individual single character, such as the letters a, b, and so on. At this point nothing has been combined, so the vocabulary is just the alphabet the tokenizer has to handle.

Because every character is present, the tokenizer always has a way to represent any text built from those characters, even text it has never seen before. A word the tokenizer has never encountered is not a problem at this stage, because each of its letters already has an entry.

The diagram shows the starting point in miniature. The word low is split into the characters l, o, and w, and each of those characters already has an entry, so low can be represented before a single merge has occurred. Merging only adds new entries on top of this alphabet; it never removes the ability to spell anything out letter by letter.

After the characters are in place, BPE begins merging character combinations that frequently occur together. The source gives a concrete case: the characters d and e may be merged into the subword de. This is a useful merge because de shows up in many common English words, including define, depend, made, and hidden. A single stored piece can stand in for a recurring fragment across many different words.

Notice what caused the merge. The decision does not come from treating d and e as special letters or from a hand-written rule about English spelling. It comes from how often the two characters occur together in the text. Subwords are shaped by frequency, and later they can be merged further into frequent whole words.

Now trace one merge step on the word define. Before the merge it is six separate character tokens. After the pair d and e is merged into de, the same word is five tokens. The diagram below shows that change.

The before row has six tokens and the after row has five. The pair de now appears as one token, and every other letter is unchanged. The merge did not change the meaning of define to the tokenizer; it only changed how many pieces the word is cut into. The same reasoning applies to made, which becomes m, a, de, and to hidden, which gains a single de token where the d and e were.

The source states that the merges are determined by a frequency cutoff. A cutoff is a threshold: a combination of characters or subwords has to occur often enough to pass it before it is merged. This is what turns the general idea of merging frequent things into a concrete rule. Without a cutoff, the tokenizer would have no stated criterion for deciding when a combination is common enough to deserve its own entry.

Read the flowchart as a per-pair test. A candidate pair is counted across the text, and the count is compared with the cutoff. If the pair passes, it is merged into a subword. If it falls short, it is skipped and remains as the smaller pieces it was built from. Each candidate is judged on its own count, so the same rule applies to a pair of letters and to a pair of longer subwords.

The cutoff also explains why the final vocabulary is a mix of sizes. Combinations that clear the threshold become subwords or even whole words, while combinations that fall short remain as smaller pieces. The cutoff therefore controls the balance between compact pieces and longer, more specific units.

Here is what the construction gives you when a word arrives that the vocabulary does not hold whole. A common word that cleared the cutoff can be stored directly and become a single token. A rare or unseen word is instead cut into whatever pieces the vocabulary does contain, which can be subwords, and in the worst case individual characters. Because the characters are always present, the word is still representable.

The rare word dewed is an illustrative case, not a word the source discusses. Suppose the vocabulary holds the characters d, e, and w, and the merged subword de. The tokenizer can cover dewed with de, then w, then the single characters e and d, so the word is represented without any entry for dewed itself. The common word define is a different story: it can be a single token only because it appeared often enough to be stored directly.