Foundation Models and Demonstration Data

Imagine a language model that has become very good at producing fluent, helpful text. It can explain a tax form, draft a cover letter, or summarize a long report. Now ask a harder question: is that model's skill enough to decide what it should do with its abilities? The same fluency that helps someone write a clear apology can also help someone write a convincing scam. Skill does not answer the question of direction.

This is the starting point for preference finetuning. Once a model is good at producing helpful text, the open question becomes which kinds of responses it should produce and which it should decline. This article follows a common path for beginners: we begin with a belief that seems reasonable, test it against a concrete question, and see where it breaks down. Along the way you will meet RLHF, the earliest successful preference finetuning algorithm, and its reward model component.

The key limit is that demonstration data does not, by itself, teach the model which conversations it should hold. The source is direct about this: the skill and the judgment are different things, and demonstrations supply only the skill. A model trained on excellent examples can write well in many situations, including situations where it should not be writing at all. Nothing in the examples tells it where the boundary lies.

The diagram below separates the two things a learner might confuse. On one side is what demonstration data provides. On the other is the target that preference finetuning is trying to reach. Notice that the second column depends on an assumption that the first column never has to make.

A tempting assumption goes like this: people broadly agree on what is good, so if we collect enough good behavior and teach it to a model, the model will simply behave well. Under this view, human preference is a single target that can be copied into a machine. The goal of preference finetuning is to get AI models to behave according to human preference, and this tempting view makes that goal sound easy.

The source calls this goal ambitious, if not impossible. The reason is that the goal rests on two assumptions. The first is that universal human preference exists. The second is that it is possible to embed that preference into an AI system. Both assumptions have to hold for the goal to be reachable as stated.

The first assumption is the harder one to justify. People come from different backgrounds and often disagree about what is good. Preference finetuning therefore has to reduce many different views to something a model can be trained on. Where that reduction loses information, the model is learning a simplified target, not the full range of what people actually prefer.

One natural reaction to controversy is to make the model cautious about every sensitive subject. This sounds safe, but the source points out that it has its own cost. Whatever a model says about a controversial issue, some users will be upset. If the model is censored too much, it may become boring and drive users away. The source also observes that fear of inappropriate responses can stop companies from releasing their applications to users at all.

The opposite extreme is a model that is too permissive. Because the same capability that helps people accomplish useful things can help them accomplish harmful things, a model that answers everything without judgment can be used to cause harm. The two extremes are not symmetric errors that can be fixed with one simple rule. Each has a cost, and the diagram shows where a useful middle ground sits between them.

The source says RLHF consists of two parts. The first part is to train a reward model that scores the foundation model's outputs. Read that part carefully for direction of information. The foundation model produces outputs. The reward model takes those outputs and assigns a score to each one. Trace this flow step by step.

The pipeline stops at the score because that is where the source's description of the first part stops. It does not name the second part of RLHF, so this article does not describe it. What you can take from the first part is that the reward model turns each foundation model output into a score, and that score is what carries the preference goal into training.