Understanding Training Data for Large Language Models
The training amount is a separate figure that must be stated on its own terms.
Two Numbers That Seem to Conflict
Imagine you read that a language model's training dataset contains 499 billion tokens, and then you read that the same model was trained on 300 billion tokens. It is tempting to conclude that one of these numbers must be wrong. Neither is necessarily wrong. They describe different things: one is the size of the dataset as listed in a table, and the other is the amount of that material the model actually consumed during training.
This article works through the GPT-3 case from the source material. The goal is to learn how to read a dataset summary carefully, separate the listed total from the amount used, and place the numbers in context by comparing them with another large dataset and with the sources that later models added.
What do you think happens?
If a dataset table adds up to 499 billion tokens, did the model see all 499 billion during training?
Reveal answer
No, the table total and the training amount can be different numbers
The GPT-3 table's subsets total 499 billion tokens, but the model was trained on 300 billion tokens. The source does not explain the difference, so the table total cannot be read as the amount used.
Reading the GPT-3 Dataset Table
The GPT-3 paper presents its dataset in a table. The table has a proportions column, which accounts for the full sampled data at 100 percent, with small adjustments for rounding errors. It also has a Number of Tokens column. The subsets listed in that column add up to 499 billion tokens.
The important point is where the 499 billion figure comes from. It is a sum of the subsets listed in the Number of Tokens column. It describes the dataset as the table lists it. It does not, by itself, describe how much of that dataset the training run used.
Listed Total Versus Amount Trained
Although the table's subsets total 499 billion tokens, the model was trained on only 300 billion tokens. The source is direct about what it does not know: the authors of the GPT-3 paper did not specify why the model was not trained on all 499 billion tokens. The accurate statement is therefore that the training run used 300 billion tokens out of a dataset total of 499 billion, and the reason for the difference is not given in the source.
Relating the Two GPT-3 Figures
The table total is 499 billion tokens and the training run used 300 billion tokens. What can you state with confidence, and what can you not state?
- 1Identify what each number describes
499 billion is the sum of the listed subsets, which is a property of the table. 300 billion is the amount used in training, which is a property of the training run.
- 2State the relationship in plain words
The training run used 300 billion of the 499 billion tokens listed. That is a valid statement from the source.
- 3Check what the source does not say
The source does not explain why the remaining tokens were not used. Any explanation you add would go beyond the source.
- 4Note the plain arithmetic
300 divided by 499 is roughly 0.60, so the training figure is about 60 percent of the listed total. This is arithmetic on two source numbers, not a reason for the gap.
The training run used 300 billion of the 499 billion listed tokens. The reason for the gap is not stated in the source.
Putting 499 Billion Tokens in Scale
To understand how large 499 billion tokens is, it helps to have a reference. The source offers the CommonCrawl dataset for this purpose. CommonCrawl alone consists of 410 billion tokens and requires about 570 GB of storage. That gives a sense of scale: one dataset of this kind is already a very large body of text.
Notice that CommonCrawl's 410 billion tokens is smaller than the 499 billion total in the GPT-3 table, but larger than the 300 billion tokens the model was trained on. The comparison is useful for scale. The source does not say how CommonCrawl relates to the GPT-3 table beyond these figures, so you should not infer more than the numbers show.
What LLaMA Added to Its Training Scope
The source also explains that later iterations of models like GPT-3 have expanded their training scope. Meta's LLaMA is the example given. Its training scope includes additional data sources beyond the general web text that dominates earlier datasets. The source names two of them, each with a storage size.
The two additional sources are much smaller in storage than the CommonCrawl dataset's 570 GB, at 92 GB for the Arxiv research papers and 78 GB for the code-related StackExchange questions and answers. The source does not give their token counts. These figures should be read as storage sizes, not as token totals that can be placed in the same column as the GPT-3 numbers.
Mistakes When Reading Dataset Tables
Common mistake Treating the table total as the amount the model was trained on
What it looks likeReading the 499 billion total and concluding that GPT-3 was trained on 499 billion tokens.
Why it failsThe total is the sum of the listed subsets. The source states that the model was trained on 300 billion tokens.
Do this insteadReport the table total as the listed size and report the 300 billion figure separately as the training amount.
Common mistake Assuming one of two conflicting numbers must be wrong
What it looks likeDeciding that either the 499 billion or the 300 billion figure is an error.
Why it failsThe two figures describe different things, so both can be accurate.
Do this insteadAsk what each number measures before deciding whether they conflict.
Common mistake Comparing token counts with gigabyte sizes directly
What it looks likeSaying that 410 billion tokens is larger than 570 GB because 410 is smaller than 570 but the token figure is the bigger one.
Why it failsTokens count units of text and gigabytes measure storage. They are not on the same scale.
Do this insteadCompare token counts with token counts and storage sizes with storage sizes.
Common mistake Inventing a reason for the gap
What it looks likeStating that the remaining 199 billion tokens were removed for quality reasons.
Why it failsThe source says the authors did not specify why the model was not trained on all 499 billion tokens.
Do this insteadState that the source does not give the reason for the difference.
Checking Your Reading of a Dataset Summary
A dataset summary lists three subsets whose token counts add up to 499 billion. It also says the model was trained on 300 billion tokens. Another line gives a storage size of 570 GB for one subset. Write two sentences: one that correctly states what the 499 billion figure describes, and one that correctly states how the 570 GB figure should be compared with the token counts.
Hints
- Ask whether the 499 billion figure is a sum of listed subsets or an amount consumed during training.
- Ask whether gigabytes and tokens measure the same kind of quantity.
Summary of Reading Training Data Figures
- The GPT-3 subsets listed in the table total 499 billion tokens, but the model was trained on 300 billion tokens, and the source does not explain why.
- CommonCrawl alone is 410 billion tokens and about 570 GB, which gives a sense of scale.
- LLaMA's training scope added sources such as Arxiv research papers (92 GB) and code-related StackExchange Q&As (78 GB), reported as storage sizes rather than token counts.
- Separate listed totals from the amount used in training, and never compare token counts directly with gigabyte sizes.
At a glance
- The 499 billion token figure is the sum of the subsets listed in the GPT-3 table, while the model was trained on 300 billion tokens.
- The source does not explain why the training run did not use all 499 billion listed tokens, so that reason must not be invented.
- CommonCrawl alone is 410 billion tokens and about 570 GB, a useful reference for scale.
- LLaMA added sources such as Arxiv research papers (92 GB) and code-related StackExchange Q&As (78 GB), given as storage sizes rather than token totals.
- Token counts and gigabyte sizes measure different things and should not be compared directly.