Tokenization: Converting Text into Tokens

When people describe a large language model, they usually talk about its size in parameters or about the amount of computing used to build it. A second question is just as important: what did the model see during training? Before a model can learn anything, someone has to decide which text goes into its training collection, how much of each kind of text is included, and how that collection is measured.

The GPT-3 model offers a concrete case study. Its pretraining collection was assembled from several sources, and the way those sources were combined says a great deal about what the model learned to imitate. We will follow one composition table through three questions: what unit the table measures the data in, where the data came from, and how much of it the model actually consumed. Each question changes how you read the numbers, so we will take them in order.

The step that turns raw text into these units is called tokenization: converting text into tokens. The direction of the transformation is the key idea. Human-readable text goes in, a sequence of token units comes out, and only that sequence is what the model reads. The detailed mechanics of tokenization are deferred to a later chapter, so for now treat it as a conversion step whose output is the unit of measurement for the whole dataset.

A useful rule of thumb is that the number of tokens in a text is roughly equal to the number of words plus the punctuation characters. The rule is approximate, so read token counts as estimates of how much text there is, not as exact word counts.

Why measure in tokens rather than documents? Documents vary enormously in length. A single long article and a single short note are each one document, yet they contain very different amounts of text. Counting tokens puts every piece of text on a common scale, so two collections can be compared by how much text they actually hold.

The GPT-3 pretraining collection drew on five named datasets. They fall into two broad families. Some are web crawl data, gathered by automatically collecting pages from the internet. Others are internet-based book collections or high-quality text. The composition table lists each dataset with a short description, its size in tokens, and its share of the training data.

CommonCrawl (filtered) contributes the biggest slice of the training mix at 60 percent. The two book collections are given equal weight at 8 percent each, even though their token counts can differ. That is why the two columns of the table must be read side by side. A large token count does not automatically mean a large share of the mix, and a share describes proportion, not raw size.

The composition table reports a total of 499 billion tokens. The model, however, was trained on 300 billion tokens. These are two different numbers answering two different questions. The first describes what the collection contains. The second describes what a training run consumed.

This distinction matters whenever you compare datasets from published papers. A figure describing what a collection contains is a different claim from a figure describing what a training run consumed. When you read a composition table, ask which of these two numbers the author is reporting.