Comparing Dataset Sizes Across Language Models

The GPT-3 paper presents its dataset in a table, and reading that table carefully is the first step.

Programming

Two Numbers That Seem to Conflict

Suppose you read that a language model's training dataset contains 499 billion tokens, and then you read that the same model was trained on 300 billion tokens. It is tempting to conclude that one of these numbers must be wrong. The source material shows that neither number is necessarily wrong. They describe different things: one is the size of the dataset as listed in a table, and the other is the amount of that material the model actually consumed during training.

What do you think happens?

Do the table's listed subsets add up to the 300 billion tokens used in training?

Yes, the table total and the training amount are the same
No, the table total is larger than the training amount
No, the table total is smaller than the training amount

Reveal answer

Answer:

No, the table total is larger than the training amount. The subsets listed in the table add up to 499 billion tokens, while the model was trained on 300 billion tokens.

The two figures answer different questions. The table total describes the dataset as listed, and the training figure describes how much of that material the model actually used.

Reading the GPT-3 Dataset Table

The GPT-3 paper presents its dataset in a table, and reading that table carefully is the first step. The table has a proportions column, which accounts for the full sampled data at 100 percent, with small adjustments for rounding errors. The table also has a Number of Tokens column. The subsets listed in that Number of Tokens column add up to 499 billion tokens.

Which figure is the listed total of the table's subsets, and which figure is the amount the model was trained on?

Worked example

Turning a Table Into a Training Statement

You are given a dataset table whose Number of Tokens subsets sum to 499 billion, and a note that the model was trained on 300 billion tokens. Write an accurate statement that uses both figures.

1
Identify the listed total
Add the subsets in the Number of Tokens column. The sum is 499 billion tokens. This describes the dataset as listed.
2
Identify the training amount
The model was trained on 300 billion tokens. This describes how much of the material was consumed during training.
3
Check whether the two are the same quantity
They are not the same quantity, because one is a listed total and the other is an amount used in training.
4
Check whether the source explains the gap
The source states that the authors did not specify why the model was not trained on all 499 billion tokens, so the reason is not given.

Result

The training run used 300 billion tokens out of a dataset total of 499 billion tokens. The source does not explain why the remaining material was not used.

What the Gap Does and Does Not Tell You

When the table total is larger than the training figure, the difference is real information about the two numbers, but it is not an explanation of the difference. The source is direct about what it does not know. The authors of the GPT-3 paper did not specify why the model was not trained on all 499 billion tokens. A careful reader therefore states only what the numbers show: the listed dataset is 499 billion tokens, and the training run used 300 billion tokens.

Placing 410 Billion Tokens in Context

To understand how large 499 billion tokens is, it helps to have a reference point. The source offers the CommonCrawl dataset for this purpose. CommonCrawl alone consists of 410 billion tokens and requires about 570 GB of storage. That gives a sense of scale: one dataset of this kind is already a very large body of text.

CommonCrawl is measured in tokens in one column and in storage in another, while the GPT-3 training run is measured only in tokens.

Notice that the CommonCrawl figure of 410 billion tokens is smaller than the 499 billion total in the GPT-3 table, but larger than the 300 billion tokens the model was trained on. This comparison is useful for scale. The source does not say how CommonCrawl relates to the GPT-3 table beyond these figures, so you should not infer more than the numbers show.

Because 300 billion is a little under three-quarters of 410 billion, the training run used a substantial share of a CommonCrawl-sized body of text. That is a derived comparison from the two token counts, not a figure the source states directly.

Later Models Widening the Training Scope

The source explains that later iterations of models like GPT-3 expanded their training scope. Meta's LLaMA is the example given. Its training scope includes additional data sources beyond the general web text that dominates earlier datasets. The source names two of them, and each comes with a storage size.

The two additional sources are much smaller in storage than the CommonCrawl dataset.

The first added source is Arxiv research papers, at 92 GB. The second is StackExchange's code-related questions and answers, at 78 GB. Both are much smaller in storage than CommonCrawl's 570 GB. The source does not give their token counts, so these figures must be read as storage sizes. They cannot be placed in the same column as the GPT-3 token figures.

Mistakes When Reading Dataset Tables

Common mistake
Treating the table total as the amount of data the model was trained on
What it looks like
Reading the 499 billion total and concluding that the model learned from 499 billion tokens.
Why it fails
The table total is the sum of the listed subsets. The source states that the model was trained on 300 billion tokens.
Do this instead
Report the listed total and the training amount separately, and name each one for what it measures.
Common mistake
Deciding that one of two conflicting figures must be wrong
What it looks like
Seeing 499 billion and 300 billion together and marking one as an error.
Why it fails
The two numbers describe different things, a listed dataset and an amount used in training.
Do this instead
Check what each figure measures before deciding whether they conflict.
Common mistake
Inventing a reason for the gap
What it looks like
Saying the model skipped the rest because that material was not useful.
Why it fails
The source states that the authors did not specify why the model was not trained on all 499 billion tokens.
Do this instead
State that the reason is not given in the source.
Common mistake
Comparing token counts directly with gigabyte sizes
What it looks like
Setting 410 billion tokens beside 570 GB and treating them as the same kind of quantity.
Why it fails
Tokens count units of text, while gigabytes measure storage. The source gives no conversion between them.
Do this instead
Keep token figures and storage figures in separate columns, and do not convert one into the other without a stated conversion.
Common mistake
Placing LLaMA's source sizes in a token column because they appear next to token figures in the same discussion

Checking Your Reading of a Dataset Summary

MEDIUM

A colleague writes: 'CommonCrawl is 570 billion tokens, so it is bigger than the GPT-3 table total.' Using only the figures in this article, identify what is wrong with this statement and rewrite it correctly.

Hints

Check which unit the 570 figure uses. The source gives it as storage.
Check which figure the source gives as CommonCrawl's token count.
Compare the token counts only with other token counts.

Your attempt (not saved)

EASY

Write two sentences describing the GPT-3 dataset. The first should give the listed total from the table. The second should give the amount used for training. Do not offer a reason for the difference.