Aligning Models With Human Preferences

For several years, the simplest way to make a language model better was to make it bigger. Each jump of roughly one order of magnitude in parameter count brought a jump in performance. The earliest model in this story, GPT-1, had about 117 million parameters. GPT-2 had 1.5 billion, and GPT-3 had 175 billion. Going from GPT-1 to GPT-3 is a three-orders-of-magnitude increase in size, all within the years 2018 to 2021.

That history raises the natural question: can this keep going? This article is about why it cannot continue without limit. Before reading on, try the prediction below, because the answer depends on taking the trend seriously one step at a time.

Think of the growth as a sequence of checkpoints. The first checkpoint is GPT-1 at about 117 million parameters. The second is GPT-2 at 1.5 billion. The third is GPT-3 at 175 billion. Each checkpoint is a much larger model than the one before it, and all three fall inside a window of roughly three years.

Notice what the timeline can and cannot tell you. It records the growth that has already happened. It does not tell you what happens beyond GPT-3. The source treats both possible futures, continued improvement and a plateau at much larger sizes, as open questions, and it does not claim to know the answer.

A useful way to reason about scaling is to follow the size of the model step by step, one order of magnitude at a time, and ask what each step requires. Rather than jumping straight to a huge number, you move through the sequence and check what each new size would demand.

The source identifies two bottlenecks that are already visible: the supply of training data and the supply of electricity. They are discussed separately because they come from different places. One comes from the information that people produce. The other comes from the physical infrastructure needed to run training.

The source describes a third concern that follows from the data bottleneck. The internet is being rapidly populated with data generated by AI models. If companies keep using internet data to train future models, those new models will be partly trained on AI-generated text. Each generation can end up learning from the output of the generation before it.

Follow the cycle around. Each pass through the loop adds more generated text to the data the next model learns from. The concern is that this recursive training may cause new models to gradually forget the original data patterns, which would degrade their performance over time. The source attributes this concern to Shumailov and colleagues, and it adds that the actual impact of AI-generated data on models is more nuanced, a point it takes up in a later chapter.

The lesson from this example is about provenance. If the web contains a large amount of model output, a model trained on web data may absorb that output even when no one decided to include it. This is why the recursive loop matters: the contamination can happen through ordinary data collection.

A related open problem is model forgetting. The source states the research question directly: how do you make a model forget specific information it learned during training? The difficulty is that removing something from the public web does not automatically remove it from a model that already learned it.