Language Modeling Fundamentals

Imagine you want to teach a model to predict the next word in a sentence. The most direct approach is to collect many examples and have people write down the correct answer for each one. That works, but it is slow, expensive, and hard to scale. Every new task seems to demand a new round of human annotation. Researchers call this the data labeling bottleneck, and it is one of the main reasons machine learning projects stall before they reach useful performance.

Language modeling offers a way around this bottleneck. The model does not need a person to write down the answers, because the raw text already contains them. In this article you will see how that works, follow one sentence step by step as it becomes training data, and look at why bigger models ask for bigger datasets.

Self-supervision is an approach that sidesteps much of the labeling cost. Instead of asking a human to attach an explicit label to each example, the model is set up so that it can work out the labels from the input data itself. The raw material already contains the answers. The training process simply needs to be arranged so that those answers are revealed to the model in the right way.

The key idea is to split each piece of input into two roles. One part serves as the context the model is allowed to look at. The other part serves as the target that the model must predict. Because the target comes from the input itself, no separate annotation step is required. The same text that feeds the model also tells you what the correct output should have been.

Language modeling fits this pattern naturally. Each input sequence provides two things at once: the tokens the model should predict, and the surrounding tokens the model can use as context when making that prediction. Nobody needs to write the answers down, because the sequence already contains them.

Before working through a new sentence, consider the example this material uses: the sentence 'I love street food.' is reported to produce six training samples from its own text. Each sample pairs some preceding material with the piece that comes next, which is what makes the sentence useful without any labels being added.

To see the mechanism without relying on that count, try a different sentence. Take 'The cat sat down.' and treat each word and the final period as a token. Starting from the first token, every position after the first gives a sample: the words before it form the context, and the current word is the target. Work through the trace below, then compare it with the diagram that follows.

A second question follows naturally once models can be trained without labels: how much data does a model need? The size of a model changes what counts as small or large. At the time of writing, a model with 100 billion parameters was considered large, and the source expects that such a figure may one day seem modest. Size is therefore relative, but the relationship between size and data is what matters here.

Larger models have more capacity to learn. Capacity is the room a model has to absorb patterns from its training material. A model with more capacity can represent more detail, but it can only use that room if it is given enough examples to fill it. Without enough data, the extra capacity has nothing to learn from, so it contributes little to performance. For that reason, larger models need more training data to reach their best results.

Training a large model on a small dataset wastes compute that a smaller model could have used well. The large model spends its training effort on a dataset that cannot exercise most of its capacity, while a smaller model matched to the same data would have put the same compute to good use.