Training Data Sources for Modern LLMs

When someone says a model was trained on a dataset, they usually mean a single number. For GPT-3 that number is less simple than it first looks. The dataset the training data was drawn from is reported at one size, and the amount the model actually consumed during training is reported at a smaller size. Reading any large-model dataset description accurately starts with noticing the gap between those two numbers.

In this article you will trace the GPT-3 figures step by step, place them next to a much larger public corpus called CommonCrawl, and then look at the extra sources that later models such as Meta's LLaMA added to their training scope.

The GPT-3 training data is described in a table that lists its component subsets together with a token count for each one. Add up the token counts in that column and the total is 499 billion tokens. This is the size of the dataset as a whole, which is the pool that training data is drawn from.

The model itself was trained on 300 billion tokens, which is less than the full 499 billion. The reported figures do not explain why training stopped short of the full pool. The accurate statement is therefore that the dataset size and the training size are two different figures, and both are reported for GPT-3.

A helpful way to keep these numbers straight is to follow them in order, the way you would track money moving from a total balance into a spending account. The walk-through below is illustrative and built from the reported GPT-3 numbers. It is not a calculation that the original reporting performs.

To judge whether 499 billion tokens is a large amount, it helps to compare it with another dataset. CommonCrawl is used for this purpose. CommonCrawl alone consists of 410 billion tokens and requires about 570 GB of storage.

Read against the GPT-3 figures, CommonCrawl's 410 billion tokens sit below the GPT-3 dataset total of 499 billion and well above the 300 billion tokens the model was trained on. CommonCrawl is therefore large on its own terms, and it is still smaller than the GPT-3 pool by the reported token count. The comparison also shows that the training count is a separate figure from either dataset total.

Later iterations of models like GPT-3 have widened their training scope. Meta's LLaMA is named as an example. Rather than relying only on a web-derived corpus, these later models include additional data sources. Two are named: Arxiv research papers, which add 92 GB to the dataset, and StackExchange's code-related questions and answers, which add 78 GB.

Notice that these additions are given in gigabytes, not tokens. The reported figures do not convert them into token counts, so 92 GB and 78 GB should not be assumed to correspond to any particular number of tokens. What is established is the direction of change: newer models drew on specialized sources such as scientific papers and programming Q&A, in addition to the kind of data the earlier GPT-3 corpus was built from.