Reading Training Data Tables Critically

When people describe a large language model, they usually start with its size in parameters or with the amount of computing used to build it. A second question is just as important: what did the model see during training? Before a model can learn anything, someone has to decide which text goes into its training collection, how much of each kind of text is included, and how that collection is measured.

The GPT-3 model is a concrete case study. Its pretraining collection was assembled from several sources, and the way those sources were combined says a great deal about what the model learned to imitate. A composition table is the compact summary of that choice, and it can look complete while still hiding two important facts. This article follows one such table through three questions: what unit the data is measured in, where the data came from, and how much of it the model actually consumed. Each question changes how you read the numbers, so we will take them in order.

The source gives a helpful rule of thumb: the number of tokens in a dataset is roughly equal to the number of words and punctuation characters in that text. The rule is approximate, so treat token counts as estimates of text volume rather than exact word counts. The step that converts raw text into these units is called tokenization, the process of turning text into tokens. The source names this step and defers its detailed mechanics to a later chapter. For now, the important idea is the direction of the transformation: human-readable text goes in, a sequence of token units comes out, and only that sequence is what the model reads.

Why are dataset sizes given in tokens rather than documents? Documents vary enormously in length. A one-line note and a long article are both one document, yet they contribute very different amounts of text. Counting documents would therefore say little about how much language the model actually read. Counting tokens tracks the volume of text more directly, which is why token counts are the common way to report dataset size. This reasoning follows from the source's point that token counts roughly track words and punctuation.

The GPT-3 pretraining collection drew on five named datasets. They fall into two broad families. Some are web crawl data, gathered by automatically collecting pages from the internet. Others are internet-based book collections or high-quality text. Each row of the composition table pairs a dataset with a short description, its size in tokens, and its share of the training data.

CommonCrawl (filtered) contributes the largest slice of the training mix at 60 percent. The two book collections are each given 8 percent, so they carry equal weight even though their token counts differ. Reading the share column next to the token column is what reveals this difference, so always check both columns rather than trusting either one alone.

The token column of the table totals 499 billion tokens. That is the size of the collection that was assembled. The model, however, was trained on 300 billion tokens. These are two different quantities, and the source does not explain why the difference exists. What the source does establish is that the proportion column describes the mix of the sampled data, which means the percentages apply to the sampled portion rather than automatically to the full 499 billion.

A composition table makes an internal claim: its shares should account for the whole training mix. The source notes that the proportion column sums to 100 percent once rounding is allowed for. That gives you a practical check. If you know some of the shares, subtract them from 100 to find what the remaining sources must cover.

Checking the token counts works the same way in spirit. Because the shares describe the sampled mix, a source's share should correspond to its portion of that mix rather than its raw token total in isolation. When a table gives both numbers for each row, compare them rather than reading either column alone. The source's own table is one where the two columns tell different stories, so the comparison is worth doing.