Parameters and Model Capacity

Most machine learning systems are trained on examples that someone has already marked with the correct answer. A photo labeled as a cat, or a review labeled as positive, gives the learner something specific to aim for. Preparing those labels takes effort, and that effort limits how much training data can be collected.

Language models follow a different path. They learn from ordinary text sequences, and no one has to attach a label to each one. This article follows that idea from the ground up. We will look at where the training text comes from, how the size of a model is counted, why that count matters, and why a bigger model changes how much data it needs. Along the way we will see that the word large is a moving target.

The reason this matters is availability. Text sequences are everywhere: books, blog posts, articles, and Reddit comments. Because the raw material is so plentiful and needs no labeling step, it is possible to assemble a very large training set. Scale in the training data is what allows a language model to grow into what we call an LLM.

The central question is how a plain sentence can become training examples when nobody annotated it. The answer is that the text itself supplies the target. Take a sentence and slide along it: at each position, the words seen so far are the input, and the word that comes next is the target the model should learn to predict. No human had to write that target down. It is already sitting in the sentence.

A parameter is therefore not something you type in before training. It starts with some value, and training adjusts it again and again as the model sees text. When training finishes, the collection of settled values is the model. Using the trained model means reading those fixed values and applying them to new input.

Before asking how large a language model is, we need a way to measure size. A model's size is typically measured by its number of parameters. Counting parameters gives a single number that describes how much the model has to adjust during training.

Parameters matter because they set how much a model can absorb. In general, a model with more parameters has a greater capacity to learn the behaviors we want. Capacity is the room a model has to learn; a model with more parameters has more room.

The source gives a concrete history that shows how quickly the idea of a large model has shifted. Each point on the timeline below records a model and its parameter count at the time it appeared, following the milestones the source mentions.

The lesson is that large describes a moving reference point, not a fixed size. The same 117 million parameters that counted as large in 2018 was downgraded to small once GPT-2 arrived with 1.5 billion. Any label of large should therefore be read against the models that exist at the time.

The source raises a question that is usually taken for granted: why do larger models need more data? Its answer starts with capacity. Larger models have more capacity to learn, and that extra capacity only pays off when there is enough training data to fill it.

The right-hand column is not a separate rule about one particular model. It follows directly from capacity. The more a model can learn, the more material it needs in order to reach its best performance. When a large model is trained on a small dataset, the training effort is spent without the capacity being put to full use, which is why the source describes the result as wasted compute.