Training Data for Language Models

Most people picture machine learning as a teacher with an answer key. You show a program thousands of photos, someone has marked each photo with what it contains, and the program learns to match pictures to those marks. The marks are called labels, and producing them is often slow and expensive.

Language models such as GPT face a serious version of this problem. Their training material is written text. Text is abundant, but nobody has attached a labeled answer to each sentence. So the question is how a model can still learn from such text, and why that makes it practical to train on very large collections of unlabeled writing. This article answers both parts, starting with a small experiment you can run in your head.

The key idea is that a text already contains information about its own structure. Words appear in a particular order, and each word sits in a particular place relative to the words around it. Self-supervised learning exploits that order. Instead of asking a human to supply the answer for each piece of text, the training procedure reads the answer straight out of the text.

Put simply, the data supplies both the question and the answer key. The earlier words are the question the model sees, and the word that really follows them is the answer it is checked against. Because the answer is already present in the data, no separate annotation step is needed.

The specific technique that GPT relies on is next-word prediction. It is a task in which the model is asked to predict the next word in a sentence or document. The next word that actually appears in the text serves as the label the model is supposed to predict.

This gives a simple rule that can be applied at every position in the text. Take the words up to some point, call them the input, and look at the word that follows. That following word is the label for that position. Repeat the rule at each position, and the text generates its own training examples.

The sentence 'The cat sat on the mat' is a generated example, not taken from the source, but it follows the same rule. Read it from left to right. Each diagram cell below shows one word of the sentence and the label that the rule assigns to it. The diagram is an index map: position, word, and the word that follows.

The sentence-tracing process can be seen as a window that moves across the text. At each step the window covers the words seen so far, which form the input. The word just after the window is the target the model must predict. Then the window grows by one word, and the next target is the word after that.

Notice where each target comes from. It is never supplied by hand. It is simply the next word already sitting in the text, waiting to be read. The model sees the window, produces a guess for the target, and the actual word from the text is the reference used for comparison.

The difference between the two approaches lies in where the answer comes from. In the photo example, people write each label after the data exists. In self-supervised training, the label is already part of the data, because it is the word that follows.

Because labels are generated as the training process reads through the text, one position at a time, any ordinary body of text can serve as training material without being prepared first. There is no need to annotate it in advance. A collection of writing becomes training data simply by being read.

The practical payoff is that self-labeling removes the bottleneck of annotation. The source states directly that next-word prediction makes it possible to use massive unlabeled text datasets to train LLMs such as GPT. The amount of training material is limited by the text available, rather than by how much human labeling time can be paid for.