Tokenization Basics
BPE is a tokenization scheme. It describes how text is split into tokens. It is not the language model itself.
Why Text Must Be Broken Apart
A language model cannot learn directly from a sentence as one undivided block. Before a model can learn anything from text, that text has to be broken into pieces. Those pieces are called tokens, and the process of producing them is called tokenization.
This article focuses on one particular tokenization scheme, byte pair encoding, usually abbreviated BPE. Keep the central question in mind as you read: when text goes into a system, what comes out, and who is responsible for each step?
Following Text Through the Pipeline
Start by tracing the flow of text rather than the inner details of any one step. Raw text enters a BPE tokenizer. The tokenizer produces tokens. Those tokens are part of what the language model learns from during training. The diagram below shows where BPE sits in this flow.
Notice what the diagram is for. It shows the order of roles, not a detailed mechanism. It does not reveal how the tokens are formed. That detail belongs to the next stage of study. For now, the important point is that the tokenizer sits between raw text and the model.
Defining Byte Pair Encoding
Byte pair encoding (BPE) is a tokenization scheme, meaning a method for splitting text into tokens.
BPE belongs to the family of tokenization methods. It is presented as more sophisticated than the simpler tokenization approaches that come before it. The word sophisticated signals that BPE is not just a basic way of cutting text apart. It is built on a specific concept, and that concept is what separates it from those simpler schemes.
BPE is a tokenization scheme. It describes how text is split into tokens. It is not the language model itself.
Where the Tokenizer Ends and the Model Begins
Two parts of the pipeline are easy to confuse: the tokenizer and the language model. The tokenizer is the component that splits text into tokens. The language model is the component that is trained on those tokens. The tokenizer produces the material; the model learns from it.
| Component | Role in the pipeline | What it produces or uses |
|---|---|---|
| Tokenizer (BPE) | Splits raw text into tokens | Tokens |
| Language model | Is trained using the tokens | A model trained on those tokens |
Each component has a distinct job in the pipeline.
BPE in Training GPT-2, GPT-3, and ChatGPT
BPE matters because it was used in practice, not only in theory. A BPE tokenizer was used to train large language models such as GPT-2, GPT-3, and the original model used in ChatGPT. BPE was therefore part of the pipeline that prepared text for training some of the best-known language models.
Take the three named models as a group. Each one was trained with a BPE tokenizer in its pipeline. The tokenizer did its job before training began: text was split into tokens, and those tokens were what the model learned from. The same tokenization scheme appears across all three, which is what is carried from one model to the next in this pipeline.
Tracing a Pipeline Step by Step
Classifying each stage of the pipeline
A training pipeline takes a sentence, splits it into tokens with a BPE tokenizer, and then trains a language model on those tokens. Identify the role of each stage.
- 1Find the raw input
The sentence is the raw text. It enters the pipeline before anything is split.
- 2Identify the tokenizer
The BPE tokenizer is the component that splits the text. Its output is tokens.
- 3Identify the model
The language model is trained on the tokens. It is a consumer of the tokenizer's output, not the splitter itself.
- 4Check the dependency
The model depends on the tokens. The tokenizer does not depend on the model's training to split text.
Raw sentence, then BPE tokenizer producing tokens, then the language model trained on those tokens.
Mistakes When Reading the Pipeline
Common mistake Treating the tokenizer and the language model as the same thing
What it looks likeSaying that BPE is the model that answers questions.
Why it failsBPE is a tokenization scheme that splits text. The language model is the part that is trained on the resulting tokens.
Do this insteadDescribe BPE as the component that produces tokens, and the language model as the component trained with them.
Common mistake Assuming the diagram explains how tokens are formed
What it looks likeReading the pipeline and concluding you now know how each token is built.
Why it failsThe pipeline shows roles and order only. The merging steps are not covered at this stage.
Do this insteadUse the pipeline to understand where BPE sits, and leave the mechanism for the next stage of study.
Common mistake Reading BPE as one of the basic splitting schemes rather than a more sophisticated one
Predicting the Producer of Tokens
What do you think happens?
In the pipeline from raw text to a trained model, which component produces the tokens?
Reveal answer
The BPE tokenizer
The tokenizer splits text into tokens. The language model is trained on those tokens, so it consumes them rather than producing them.
Write two sentences. In the first, name the component that splits text into tokens. In the second, name the component that is trained on those tokens. Then explain in one sentence why the two components should not be confused.
Hints
- Look back at the pipeline diagram and follow the arrow from raw text.
- The tokenizer is the splitter. The model is the learner.
Key Takeaways
- Tokenization is the process of breaking text into pieces called tokens, which a language model needs before it can learn from text.
- Byte pair encoding (BPE) is a tokenization scheme, considered more sophisticated than the simpler schemes that come before it.
- A BPE tokenizer was used to train large language models such as GPT-2, GPT-3, and the original model used in ChatGPT.
- The tokenizer produces tokens, and the language model is trained on them. They are separate parts of the pipeline.
- The merging rules that form BPE tokens are a topic for the next stage of study.