Engineering Considerations in Training Large Models

Suppose you want a model to predict the next word in a sentence. The most direct approach is to collect many examples and have people write down the correct answer for each one. That approach works, but it is slow, expensive, and hard to scale. Every new task seems to demand a new round of human annotation. Researchers call this the data labeling bottleneck, and it is one of the main reasons machine learning projects stall before they reach useful performance.

This lesson follows one idea from start to finish. Self-supervision avoids most of that annotation cost by letting the model work out its own labels from the input data. The raw text already contains the answers. Training only has to be arranged so that those answers are revealed to the model in the right way. We will trace what happens to a single unlabeled sentence, then look at why bigger models raise the question of how much data is enough.

In manual labeling, a person looks at each example and attaches the correct output. The labels are an extra product that someone has to create, and the process has to be repeated for each new task. In self-supervision, no separate annotation step is required. The model is set up so that it can derive the correct output from the input it already has.

The difference matters for scale. Manual labeling grows with the number of examples someone has to annotate, which is why it is described as slow and hard to scale. Self-supervision draws its labels from text that is already available, so the labeling step does not have to be repeated by people for every new example.

The key idea is to split each piece of input into two roles. One part serves as the context that the model is allowed to look at. The other part serves as the target that the model must predict. Because the target comes from the input itself, no separate annotation step is needed. The same text that feeds the model also tells you what the correct output should have been.

Language modeling fits this pattern naturally. Each input sequence provides two things at once: the tokens the model should predict, and the surrounding tokens the model can use as context when making that prediction. Nobody needs to write the answers down, because the sequence already contains them.

Take the sentence 'The cat sat down.' and treat each word and the final period as a token. Starting from the first token, every position after the first gives one training sample. The words before that position form the context, and the current token is the target. Each token after the first is therefore a target exactly once.

Notice what changes from one sample to the next. The context grows by one token each time, and the target moves one position to the right. The label for each sample is simply the token that comes next in the original text. The same rule applies to any sentence, so the number of samples depends on how many tokens the sentence contains after the first one.

Once models can be trained without labels, a second question follows: how much data does a model need? The source material points out that model size changes what counts as small or large. At the time of writing, a model with 100 billion parameters was considered large, and the source expects that such a figure may one day seem modest. Size is therefore relative, but the relationship between size and data stays the same.

Capacity is only useful if it is filled. A model with more capacity can represent more detail, but it can only use that room if it is given enough examples. Without enough data, the extra capacity has nothing to learn from, so it contributes little to performance. For that reason, larger models need more training data to reach their best results.

Training is not free. When a large model is trained on a small dataset, the compute spent on it is not matched by the amount of material it can learn from. The unused part of the model's capacity sits idle, and the compute budget that paid for it is spent on a model that a smaller model could have used well.

Self-supervision removes the need for a separate annotation step by splitting each input into context and target. In language modeling, one unlabeled sentence yields a training sample for every token after the first. Larger models have more capacity, so they need more training data to use that capacity well, and training a large model on a small dataset wastes compute.