Scaling Language Models

Most machine learning systems are trained on examples that someone has marked with the correct answer. A photo labeled as a cat, or a review labeled as positive, gives the learner something specific to aim for. Preparing those labels takes effort, and that effort limits how much training data can be collected. Language models follow a different path: they learn from ordinary text sequences, and no one has to attach a label to each one.

Machine learning covers many kinds of models. Some detect objects in images, some forecast the weather, some predict stock prices, and some recommend products. Language models are one family among these, yet they became the center of the scaling effort that led to the ChatGPT moment. The reason is a property of how they can be trained: they can learn from self-supervision, while many other model types need supervision. This article explains what that property means and why it matters.

In supervised learning, you show a model examples of the behavior you want and attach a label to each one. Once trained, the model can be applied to new data. A fraud detection model, for instance, learns from transactions that are each tagged as fraud or not fraud, and then predicts the tag for transactions it has never seen. The success of many AI systems in the 2010s rested on this approach.

The weakness is cost. Obtaining labels is expensive and slow, and this is the data labeling bottleneck that limits how large a supervised dataset can grow. The bottleneck gets worse as tasks become harder. Labeling everyday objects is something most people can do without training, but labeling whether a medical scan shows signs of cancer demands expert time and is far more costly.

Self-supervision removes the need for an external answer key. Instead, the model infers the labels from the input data itself. Language modeling is a clean case of this. Every input sequence contains two things at once: the tokens the model should predict, and the surrounding tokens that serve as context for predicting them. Because the text supplies both, a large body of ordinary writing can be turned into training material without anyone annotating it.

Take a new sentence to see the mechanism: Cats chase mice. The sentence is first wrapped in a beginning marker and an end marker. It has four ordinary tokens, and each position in the sequence becomes one training sample. The learner sees the context to the left of a position and must predict the token that comes next. Watch how the context grows one token at a time.

Notice what happened. A single sentence produced five separate training samples, and no one wrote a label for any of them. A longer passage of text yields many more samples in the same way, which is why this approach can scale to very large collections of writing.

The markers at the edges of each sequence are special tokens. The beginning marker, written <BOS>, shows where a sequence starts. The end marker, written <EOS>, shows where it stops. Each marker is typically handled by the model as one token, not as a set of characters. These markers matter because a language model usually works with many sequences, and it needs a clear way to tell where one ends and the next begins.

In the sentence example, <BOS> sits at the left edge and gives the model a starting point for the first prediction. <EOS> sits at the right edge. The source notes that the end marker teaches the model when to stop. In the final sample, the context is the full sentence and the target is <EOS>, so the model is trained to predict the end of the sequence as well as the words within it.

The distinction is not whether a model is trained on raw data, since both approaches can use it. The distinction is whether the learning setup relies on labels, and where those labels come from. Use the presence or absence of labels as the key test.

Self-supervised training follows a chain. Text sequences are gathered from sources such as books, blog posts, articles, and Reddit comments. Each gathered sequence is wrapped in <BOS> and <EOS> and split into positions. At each position, the text itself supplies the target, the token that really comes next. The model makes its prediction from the context, and its parameters, which are variables updated during training, change as training runs. No step asks a person to mark what each sequence means.

Before asking how large a language model is, we need a way to measure size. A model's size is typically measured by its number of parameters. The source defines a parameter as a variable within an ML model that is updated through the training process. Each time training runs, these variables change, and the final values are what the trained model uses.

Parameters matter because they set how much a model can absorb. In general, a model with more parameters has a greater capacity to learn the behaviors we want. The source adds a necessary qualification: this is not always true. Parameter count is a useful guide, not a guarantee, so it should be read as a tendency rather than a rule.

The source gives a concrete history that shows how quickly the idea of a large model has shifted. Each milestone records a model and its parameter count at the time it appeared.

This is why the source observes that what is large today might be considered tiny tomorrow. The label is relative to the models available at the time.

The source raises a question that is usually taken for granted: why do larger models need more data? Its answer starts with capacity. Larger models have more capacity to learn, and that extra capacity only pays off when there is enough training data to fill it. A model with a large capacity but little to learn from is like a student with a big memory and only a few pages to read.

The consequence follows directly from capacity. The more a model can learn, the more material it needs to learn from in order to reach its best performance. Model size and dataset size therefore need to move together.