Tokens and How Text Is Counted

To understand how large 499 billion tokens is, it helps to have a reference point.

Programming

Two Numbers That Seem to Conflict

Suppose you read that a model's training dataset contains 499 billion tokens, and then you read that the same model was trained on 300 billion tokens. It is tempting to conclude that one of these numbers must be wrong. Neither is necessarily wrong. They describe different things: one is the size of the dataset as listed in a table, and the other is the amount of that material the model actually consumed during training.

What do you think happens?

The GPT-3 dataset table's subsets total 499 billion tokens, and the model was trained on 300 billion tokens. Are these the same amount?

Yes, they are the same amount
No, they describe different things

Reveal answer

Answer:

No. The 499 billion figure is the listed total of the dataset table, and the 300 billion figure is the amount the model was trained on.

The GPT-3 paper does not specify why the model was not trained on all 499 billion tokens, so the reason for the difference is not given in the source.

This article follows the GPT-3 case step by step. The goal is to learn how to read a dataset summary carefully, separate the listed total from the amount used, and place both numbers in context by comparing them with another large dataset and with the sources that later models added.

Reading the GPT-3 Dataset Table

The GPT-3 paper presents its dataset in a table. The table has two columns that matter for reading it. The first is a proportions column, which accounts for the full sampled data at 100 percent, with small adjustments for rounding errors. The second is a Number of Tokens column, which lists the token count for each subset of the data.

The subsets listed in the Number of Tokens column add up to 499 billion tokens. This is the total you read in the table. It describes the dataset as listed, and it is a sum of the rows, not a report of how much text the training run read.

Which number describes the dataset as listed, and which describes the amount the model was trained on? The listed total is larger than the training amount, and the source does not explain the gap.

Why 499 Billion Is Not 300 Billion

Although the table's subsets total 499 billion tokens, the model was trained on only 300 billion tokens. The source is direct about what it does not know: the authors of the GPT-3 paper did not specify why the model was not trained on all 499 billion tokens.

Key point

The accurate statement is that the training run used 300 billion tokens out of a dataset total of 499 billion. The reason for the difference is not given in the source, so you should not invent one.

Worked example

Stating the GPT-3 Figures Correctly

You read that the GPT-3 table totals 499 billion tokens and that the model was trained on 300 billion tokens. Write a correct one-sentence summary of these two numbers.

1
Identify the listed total
The subsets in the Number of Tokens column add up to 499 billion tokens. This is the dataset size as listed.
2
Identify the training amount
The model was trained on 300 billion tokens. This is the amount consumed during the training run.
3
Compare the two without assuming a cause
The training amount is smaller than the listed total. The source does not say why, so the summary should state the difference without explaining it.

Result

The GPT-3 dataset table lists 499 billion tokens in total, and the model was trained on 300 billion of them; the source does not explain the difference.

Measuring Scale with CommonCrawl

To understand how large 499 billion tokens is, it helps to have a reference point. The source offers the CommonCrawl dataset for this purpose. CommonCrawl alone consists of 410 billion tokens and requires about 570 GB of storage. That gives a sense of scale: one dataset of this kind is already a very large body of text.

Compare the three token figures. CommonCrawl's 410 billion tokens is smaller than the 499 billion total in the GPT-3 table, but larger than the 300 billion tokens the model was trained on. The comparison is useful for scale, but the source does not say how CommonCrawl relates to the GPT-3 table beyond these figures. Do not infer more than the numbers show.

Tokens Versus Gigabytes

CommonCrawl appears in the source with two different units. It is measured as 410 billion tokens and as about 570 GB of storage. These are two different kinds of measurement. A token count describes how many units of text the dataset contains. A gigabyte figure describes how much storage space the data occupies. The two numbers answer different questions.

The GPT-3 token figures are shown in one column and the CommonCrawl storage size in another. Only the token column can be placed beside the GPT-3 token totals; the gigabyte figure answers a different question.

Key point

Do not compare token counts directly with gigabyte storage sizes. A figure of 570 GB cannot be read as a token total, and a token total cannot be read as a storage size. Keep each number in its own unit when you place it in a comparison.

Sources Added in Later Models

The source also explains that later iterations of models like GPT-3 have expanded their training scope. Meta's LLaMA is the example given. Its training scope includes additional data sources beyond the general web text that dominates earlier datasets. The source names two of them, and each comes with a storage size.

Two additional sources named in the source, each with its storage size. Both are much smaller than the CommonCrawl storage size of 570 GB.

The two additional sources are much smaller in storage than the CommonCrawl dataset's 570 GB. They are 92 GB for Arxiv research papers and 78 GB for StackExchange's code-related Q&As. The source does not give their token counts, so these figures should be read as storage sizes, not as token totals that can be placed in the same column as the GPT-3 numbers.

Mistakes When Reading Dataset Tables

Common mistake
Treating the table total as the amount of data the model used
What it looks like
Reading the GPT-3 table total of 499 billion tokens and concluding that the model was trained on 499 billion tokens.
Why it fails
The table total is the sum of the listed subsets. The training run used 300 billion tokens, according to the source.
Do this instead
Report the listed total and the training amount as two separate figures, and label each one.
Common mistake
Explaining the gap between 499 billion and 300 billion
What it looks like
Stating that the extra 199 billion tokens were discarded because they were low quality.
Why it fails
The source does not explain why the model was not trained on all 499 billion tokens, so any stated cause would be invented.
Do this instead
Say that the reason is not given in the source.
Common mistake
Comparing tokens with gigabytes directly
What it looks like
Deciding that 570 GB of CommonCrawl is larger than 410 billion tokens, or that 92 GB is smaller than 300 billion tokens, as if both were the same kind of number.
Why it fails
Token counts and storage sizes measure different things. The source says not to compare them directly.
Do this instead
Keep token figures beside other token figures, and storage figures beside other storage figures.
Common mistake
Placing LLaMA's source sizes in a token column
What it looks like
Adding the 92 GB Arxiv figure to a list of token totals for the GPT-3 comparison.
Why it fails
The source gives storage sizes for these additional sources and does not give their token counts.
Do this instead
Mark these values as storage sizes in any table you build.

Practice: Sorting Listed and Used Figures

EASY

A summary states: The dataset lists 410 billion tokens for CommonCrawl, the GPT-3 table totals 499 billion tokens, the model was trained on 300 billion tokens, and CommonCrawl requires about 570 GB of storage. Sort each figure into one of three groups: listed total, amount used in training, or storage size. Then state which figure cannot be placed in the same column as the token counts.

Hints

Ask whether each figure is a sum of listed subsets, a training amount, or a storage measurement.
The source says that the 570 GB figure is a storage size, not a token count.

Your attempt (not saved)

MEDIUM

Write two sentences for a reader who has only seen the GPT-3 table total. The first sentence should state the listed total. The second should state the training amount and note that the source does not explain the difference.