Splitting Text into Word and Subword Tokens

Large language model training requires text to be prepared first, and the source treats this as a necessary step.

Programming

Why Raw Text Cannot Go Straight into Training

A large language model learns from text, but the text that exists in the world is not in a form a model can use directly. Documents arrive with mixed formatting, varied spelling, and no agreed boundaries between their parts. Before that material can be used for training, it has to be prepared.

Keep one question in mind as you read this article: when a passage of text enters the preparation process, what does it look like on the way out? The answer depends on the tokenization method. We will follow that question from the general idea down to the more advanced method the source introduces, byte pair encoding.

Before reading on, take a moment to decide what you think the word 'tokenization' means. Write down one guess. You will check it against the definition in the next section.

Following Text Through the Preparation Step

The source names two related ideas that sit on either side of the preparation step. Raw text is prepared for training, and part of that work is splitting text into tokens. Those tokens can be word tokens or subword tokens. The source treats this preparation as a necessary step before training.

The diagram below shows that flow as a simple sequence. It describes only the stages the source names, not every operation a real system might perform. Read it as a state change: the text starts raw, becomes prepared, and ends as a collection of tokens. The tokens at the end are what the rest of the training process operates on. The source does not list further intermediate states, so this article does not add any.

What happens to a piece of raw text, in order, before a language model can use it for training?

The diagram is a sequence of changes, and each arrow moves the text into a new state. The first state is text that a model cannot use directly. The final state is a set of tokens. Everything between those two states is part of the preparation the source describes.

Word Tokens and Subword Tokens

Definition

A word token is a piece of text at roughly the size of one word. A subword token is smaller than a whole word, so one word may become several tokens.

The source names only these two kinds of token. It does not give a procedure for producing them, so the example that follows is a generated illustration of the idea, not a method taken from the source. Treat it as a picture of the difference, not as a rule for how any particular system splits text.

Take the rare word 'unbreakable'. Under word-level tokenization, the whole word is one token. Under subword-level tokenization, the same word is represented by several smaller pieces. The diagram below places the two side by side.

How does the same word split differently under word-level tokenization versus subword tokenization, and why does a rare word break into several pieces?

Notice that the subword version does not need a single piece to stand for the full word. A word that is less common can be built from smaller pieces, and the same pieces can be reused across many words. This is the central practical difference: word tokens match the size of words, while subword tokens are smaller than words and may combine to form them.

Tracing Byte Pair Encoding Step by Step

Byte pair encoding is presented in the source as a more advanced method of tokenizing text. It is not a separate starting point. It belongs to the same family of work as the basic splitting ideas: turning text into tokens that a model can handle during training. The word 'advanced' is the key signal that it builds on the ideas above.

The source does not walk through the mechanics of byte pair encoding, so the trace below is a generated toy illustration. It shows the general pattern of merging adjacent pieces one step at a time, starting from individual symbols. Watch how the set of pieces changes after each merge.

Which adjacent pair of symbols gets merged first, and how does the token set change after each successive merge?

In this toy trace, the first merge joins two neighboring symbols into one new piece, and the second merge joins that piece with the next symbol. Each step leaves fewer pieces than before. The example chooses one reasonable order of merges for clarity; the source does not specify the rule a real byte pair encoding procedure uses to choose which pair to merge.

Mistakes Beginners Make About Tokens

Common mistake
Assuming a token always equals a word
What it looks like
The unfamiliar word 'unbreakable' is shown as three pieces under subword tokenization, not one.
Why it fails
The source defines a word token as roughly one word, but a subword token is smaller, so one word may become several tokens.
Do this instead
Think of a word token as one possible kind of token. Subword tokens are a different kind, and a single word may map to several of them.
Common mistake
Assuming a token is always a single character
What it looks like
Reading 'unbreakable' as eleven separate letters because each character seems small.
Why it fails
The source says a subword token is smaller than a whole word. It does not say subword tokens are individual characters.
Do this instead
A subword token sits between a character and a whole word in the source's framing. It is a piece of text larger than the word's full spelling is not required to be, but it is not automatically one letter.
Common mistake
Treating byte pair encoding as an unrelated alternative that starts from scratch
What it looks like
Believing byte pair encoding has nothing to do with word or subword tokens.
Why it fails
The source presents byte pair encoding as a more advanced way of doing the same kind of work, building on the splitting ideas.
Do this instead
Read byte pair encoding as a step beyond the basic ideas, aimed at the same goal of producing tokens a model can handle.
Common mistake
Skipping preparation and feeding raw text straight into training
Why it fails
The source treats preparing text for training as a necessary step, because raw documents have mixed formatting and no agreed boundaries.
Do this instead
Always think of raw text as passing through preparation and tokenization before the training process uses it.
Common mistake
Assuming there is only one correct way to split text into tokens

Checking Your Understanding of Tokens

EASY

Using the 'unbreakable' example from earlier, explain in your own words why a word token and a subword token are different kinds of unit, and say which one a rare word is more likely to be broken into under subword tokenization.

Hints

Reread the definitions: a word token is roughly one word; a subword token is smaller than a whole word.
Ask yourself what happens to one word when the pieces are smaller than the word itself.

Your attempt (not saved)

MEDIUM

Explain why the source treats preparing raw text as a necessary step before training, and name the two kinds of token that the preparation can produce.