How Language Models Learn from Text

A large language model does not arrive knowing how language works. It learns patterns from text, and the text it learns from is its training dataset. Before asking how a model behaves, it helps to ask what it was shown. The answer shapes a great deal of what the model can do later, so the size and variety of its training material are central to the whole process, not background details.

This article follows one question through the material: why do the datasets behind popular models need to be both enormous and varied? You will see what those datasets contain, how their scale and diversity connect to the range of tasks a model can handle, and how to read a concrete example of one such dataset.

The contents of these corpora cover a vast array of topics. They also include two broad kinds of language. Natural languages are the human languages used for everyday communication. Computer languages, such as programming languages, are the other kind. A corpus that contains both gives a model exposure to the patterns of human writing and to the structured patterns of code.

Two properties define the corpus described in the reference material. The first is scale: the corpus is measured in billions of words, which is far more text than any single person could read in a lifetime. The second is breadth: the text is not limited to one subject or one kind of writing. A corpus that mixes many topics and many languages shows the model many different ways of arranging words and ideas.

Follow a single passage of training text through the pipeline in your head. It starts as raw written material inside a corpus. The model sees it as a stream of words that it uses to learn patterns. Each piece of text the model reads adds to its experience of language. Because the corpus is both large and varied, the model's experience includes many topics, many arrangements of words, and both human and programming languages.

The reference material names both GPT-like and BERT-like models as the families whose large training datasets it describes. The two families differ in how they use the same sentence during learning. A GPT-style model reads text from left to right and predicts the next token. A BERT-style model hides some tokens and fills them in using context from both sides of the hidden position.

The difference does not change the core lesson of this article. Either way, the model learns only from the text it is given. A corpus that is large and varied gives either style of model many more chances to observe how words, topics, and languages fit together.

A corpus on its own is only raw material. What gives it value is what a model can do after learning from it. The reference material states the central takeaway directly: the scale and diversity of the training dataset allow these models to perform well on diverse tasks. The tasks it names are language syntax, which concerns how words are arranged into valid structures; semantics, which concerns meaning; and context, which concerns how meaning shifts with surrounding text. Some of those tasks also require general knowledge.

Abstract descriptions are easier to trust when you can point to a real case. The reference material gives one: the dataset used for pretraining GPT-3. Pretraining is the stage where a model learns from a large corpus before it is applied to particular uses. GPT-3 matters here for a specific reason: it served as the base model for the first version of ChatGPT. That places its dataset at the foundation of a model many people encountered directly.

The composition of that dataset is summarized in Table 1.1, which the reference material places alongside its description of the training corpora. The per-source figures are not reproduced in the material available for this article, so this lesson teaches you how to read the table rather than quoting its numbers.