Preparing Text for Large Language Model Training
Large language model training requires text to be prepared first, and the source treats this as a necessary step.
Why Raw Text Is Not Ready for Training
A large language model learns from text, but the text that exists in the world is not in a form a model can use directly. Documents arrive with mixed formatting, varied spelling, and no agreed boundaries between their parts. Before that material can be used for training, it has to be prepared.
Tracing a Passage Through Preparation
The source describes preparation as a sequence of states. Raw text is prepared for training, and part of that work is splitting text into tokens, which can be word tokens or subword tokens. The diagram below shows that flow as a simple sequence. It describes only the stages named in the source, not every operation a real system might perform.
Read the diagram as a state change. The text starts raw, becomes prepared, and ends as a collection of tokens. Each arrow is a change of state, and the tokens at the end are what the rest of the training process operates on. The source does not list further intermediate states, so this article does not add any.
Word Tokens and Subword Tokens
A word token is a piece of text at the size of a word. A subword token is smaller than a whole word, so a single word may be represented by more than one token.
The source names only these two kinds of token. It does not give a procedure for producing them, so the split shown in the next diagram is a generated illustration of the idea rather than a method taken from the source. The contrast it draws is about what happens to a long or uncommon word: a word-level approach keeps it as one piece, while a subword approach can break it into smaller pieces that are each more likely to be familiar.
Splitting a long word two ways
Consider the word unbelievable. How many tokens does it become under word-level tokenization, and under a generated subword split?
- 1Word-level view
A word token is roughly one word, so unbelievable is treated as a single token.
- 2Subword view
A subword token is smaller than a word, so the same word is represented by several pieces, for example un, believ, and able in this illustration.
- 3Compare the outcomes
The word-level result has one token and the subword result has three. Each subword piece is shorter and may appear in many other words, which is the reason subword pieces are useful.
Word-level tokenization gives one token for unbelievable; the generated subword split gives three.
Byte Pair Encoding as a More Advanced Method
Byte pair encoding is presented in the source as a more advanced method of tokenizing text. The word advanced is the key signal. Byte pair encoding is not a separate starting point; it is a step beyond the basic splitting ideas above. It belongs to the same family of work: turning text into tokens that a model can handle during training.
The source does not describe the internal procedure of byte pair encoding. The general idea used by this method, which the following illustration shows, is that frequently occurring neighboring pieces are joined into a larger piece, step by step. This is a generated illustration of that idea and is not a full specification.
Common Misunderstandings About Tokens
Common mistake Assuming a token is always the same thing as a word
What it looks likeA long word may be split into several subword tokens rather than being one token.
Why it failsThe source names two kinds of token. A word token is roughly one word, but a subword token is smaller, so one word may become several tokens.
Do this insteadThink of a token as a piece of text that may be a word or a smaller part of one.
Common mistake Assuming text is ready for training as soon as it is collected
What it looks likeA document with mixed formatting and no agreed boundaries is treated as directly usable.
Why it failsThe source says text in the world is not in a form a model can use directly and must be prepared first.
Do this insteadTreat preparation, including tokenization, as a required step before training.
Common mistake Treating byte pair encoding as the starting point of tokenization
What it looks likeReading byte pair encoding as the first and only way to split text.
Why it failsThe source presents byte pair encoding as a more advanced method, a step beyond the basic splitting ideas.
Do this insteadLearn the basic word and subword ideas first, then read byte pair encoding as an extension of them.
Common mistake Assuming the source gives a full procedure for making subword tokens
What it looks likeExpecting the un, believ, able split to be the exact output of a standard method.
Why it failsThe source names the token kinds but does not describe how they are produced; the split was a generated illustration.
Do this insteadUse generated examples to build intuition, and keep the source's own claims separate from illustrations.
Practice Tracing Preparation
A document enters the preparation process as raw text. Describe, in your own words, the state it is in at each stage shown in the pipeline diagram. Then explain why a long, uncommon word might be represented by more than one token under subword tokenization but by one token under word-level tokenization.
Hints
- Check the three stages in the pipeline: raw text, prepared text, and tokens.
- Recall that a subword token is smaller than a whole word, so one word may become several tokens.
- Remember that the source does not give a procedure for producing subword tokens, so keep your answer to the conceptual difference.
Key Takeaways
- Text in the world arrives with mixed formatting, varied spelling, and no agreed boundaries, so it must be prepared before a model can be trained on it.
- Preparation includes splitting text into tokens, which can be word tokens (roughly one word each) or subword tokens (smaller pieces, so one word may become several).
- Byte pair encoding is presented in the source as a more advanced method of tokenizing text, building on the basic word and subword ideas.
- Common misunderstandings include equating tokens with words, treating raw text as directly usable, and assuming byte pair encoding is the starting point of tokenization.