How Large Language Models Are Pretrained

When people describe a large language model, they often talk about its size in parameters or about the amount of computing used to build it. A second question matters just as much: what did the model see during training? Before a model can learn anything, someone has to decide which text goes into its training collection, how much of each kind of text is included, and how that collection is measured.

The GPT-3 model gives a concrete case study. Its pretraining collection was assembled from several sources, and the way those sources were combined shows a great deal about what the model learned to imitate. This article follows one dataset table through three questions: what unit the table measures the data in, where the data came from, and how much of it the model actually consumed. Each question changes how you read the numbers, so we will take them in order.

Why count tokens instead of documents? A document can be a single sentence or a very long article, so counting documents says little about how much text a model has seen. Counting tokens measures the text itself. A collection of ten short documents and a collection of ten long ones may differ enormously in volume, and the token count captures that difference while the document count does not.

The source gives a helpful rule of thumb: the number of tokens in a dataset is roughly equal to the number of words and punctuation characters in that text. Treat this as an estimate of text volume rather than an exact word count, because the rule is approximate.

The step that turns raw text into these units is called tokenization, the process of converting text into tokens. This article does not cover its detailed mechanics, which come in a later chapter. The important idea is the direction of the transformation: human-readable text goes in, a sequence of token units comes out, and only that sequence is what the model reads.

The GPT-3 pretraining collection drew on five named datasets, which fall into two broad families. Some are web crawl data, gathered by automatically collecting pages from the internet. Others are internet-based book collections or high-quality text. The table lists each dataset with a short description, its size in tokens, and its share of the training data.

The two book collections are given equal weight at 8 percent each, even though their token counts differ. This is the key lesson of reading the table: the share column reflects the mix of the sampled data, while the token column reports sizes. Reading the two columns side by side reveals differences that either column alone would hide, so always check both.

The token column totals 499 billion, but the model was trained on 300 billion tokens. These are two different claims. The 499 billion figure describes what the collection contains, while the 300 billion figure describes what the training run consumed. A model can be trained on only part of a collection.

This distinction matters whenever you compare datasets from published papers. A figure describing what a collection contains is a different claim from a figure describing what a training run consumed. When you read a composition table, ask which of these two numbers the author is reporting.

The proportion column describes the mix of the sampled data and sums to 100 percent once rounding is allowed for. You can use that fact as a check. If the listed shares add up to something far from 100, the table is internally inconsistent, or it is describing something other than a complete mix.