Tokens and Vocabulary in Language Models

When people say a model was trained on a dataset, they usually mean a single number. For GPT-3, that number is not as simple as it first looks. The dataset the training data was drawn from is reported at one size, and the amount the model actually consumed during training is reported at a smaller size. Understanding the gap between those two numbers is the first step toward reading any large-model dataset description accurately.

In this article you will follow the GPT-3 figures from the full pool to the training run, place them beside the much larger CommonCrawl corpus, and then look at the additional data sources that later models added. Throughout, keep one question in mind: is a given figure a count of tokens, a size in gigabytes, or a share of a sample?

The GPT-3 training data is described in a table that lists its component subsets together with a count of tokens for each one. If you add up the token counts in that column, the total is 499 billion tokens. This is the size of the dataset as a whole. It is the pool that the training data was drawn from.

The model itself was trained on 300 billion tokens, which is less than the full 499 billion. The source does not give a reason for this. The authors of the GPT-3 paper did not explain why training stopped short of the full pool, so the accurate statement is that the size of the dataset and the size of the training run are two different figures. Neither number is wrong; they answer different questions.

The GPT-3 data table has two numeric columns, and they measure different things. The token column gives the size of each subset, so it tells you how big each part of the pool is. The proportions column gives each subset's share of the sampled data. Those shares add up to 100 percent of the sample, with small adjustments made for rounding errors.

In short, the proportions tell you how the sample is split across subsets, while the token column tells you how big the pool is. When you read a dataset description, check which column a number came from before you interpret it.

To judge whether 499 billion tokens is a large amount, it helps to compare it with another dataset. The source uses the CommonCrawl dataset for this purpose. CommonCrawl alone consists of 410 billion tokens and requires about 570 GB of storage.

The two measures in that sentence answer different questions. Tokens measure how much text there is in units the model reads, while gigabytes measure how much disk space the data occupies. Because the source gives CommonCrawl in both units, you can compare it with the GPT-3 figures on tokens. The source does not give a storage size for the GPT-3 dataset, so no storage comparison with GPT-3 can be made from the source.

On the token measure, the GPT-3 dataset total is the largest of the three figures, CommonCrawl is in the middle, and the GPT-3 training count is the smallest. The comparison also shows that a single public corpus can be very large on its own terms even though it is not the whole of a particular model's data.

The source notes that later iterations of models like GPT-3 have widened their training scope. Meta's LLaMA is named as an example. Rather than relying only on a web-derived corpus, these later models include additional data sources. Two are named in the source. Arxiv research papers add 92 GB to the dataset, and StackExchange code-related questions and answers add 78 GB.

What the source does establish is the direction of change. Newer models drew on specialized sources such as scientific papers and programming question-and-answer material, in addition to the kind of data that the earlier GPT-3 corpus was built from.