Reward Models in RLHF

A language model that can help people accomplish useful things can, with the same skill, help people accomplish harmful things. This two-sided quality is the starting point for preference finetuning. Capability alone does not decide what a model should do with it. Once a model is good at producing fluent, helpful text, the open question becomes which kinds of responses it should produce and which it should decline.

This article follows a path that many beginners find useful. We start with a belief that seems reasonable at first glance, test it against a concrete question, and see where it breaks down. Along the way you will meet RLHF, short for Reinforcement Learning from Human Feedback, which the source identifies as the earliest successful preference finetuning algorithm and one that remains popular today, and you will meet the reward model that is one of its two parts.

Before reading on, pause and ask yourself a question: if a model is trained on many examples of good conversations, will it know when it should refuse a request? Keep your answer in mind, because the next section tests it.

The key point is what demonstration data does not do. It does not, by itself, teach the model which conversations it should hold. The skill and the judgment are different things, and demonstrations supply only the skill. A model shown many polished examples learns how polished conversations are built. Nothing in those examples, by itself, tells the model whether a particular kind of conversation is one it should take part in.

A tempting assumption goes like this: people broadly agree on what is good, so if we collect enough good behavior and teach it to a model, the model will simply behave well. Under this view, human preference is a single target that can be copied into a machine. The belief sounds sensible, which is exactly why it is worth testing against a concrete question.

The stated goal of preference finetuning is to get AI models to behave according to human preference. The source calls this an ambitious, if not impossible, goal. The phrase matters: the goal is not presented as a solved engineering task but as something that is hard to achieve at all.

The reason the goal is so difficult is that it rests on two assumptions. The first is that universal human preference exists. The second is that it is possible to embed that preference into an AI system. Each assumption can be questioned, and the source's skepticism follows from questioning them.

A natural reaction to controversy is to make the model cautious about every sensitive subject. The source points out that this has its own cost. Whatever a model says about a controversial issue, some users will be upset. But if the model is censored too much, it may become boring and drive users away. The source also observes that fear of inappropriate responses can stop companies from releasing their applications to users at all.

The preference goal is the reason preference finetuning exists; RLHF is the concrete method used to pursue it. The source says RLHF consists of two parts. This article covers only the first. The first part is to train a reward model that scores the foundation model's outputs. The passage stops before naming the second part, so it is not described here.

What the first part makes clear is the direction of information. The foundation model produces outputs, and the reward model assigns a score to each one. The reward model sits downstream of the foundation model's generation: it does not write the response, it evaluates one.