Reward Models in Preference Finetuning

A foundation model trained on demonstration data can learn to hold a conversation, but demonstration data does not teach it which kinds of conversations it should have. A model that can help users accomplish great things can also help them accomplish terrible things. Preference finetuning is the family of methods meant to close this gap. Its purpose is to shape how a model responds so that its behavior matches what people prefer.

The hard part is that people do not always agree. Two requests make the point. If a user asks for an essay arguing that one race is inferior, or for instructions for hijacking a plane, most people agree immediately that the model should refuse. Many other requests are not that clear. Questions about abortion, gun control, the Israel-Palestine conflict, disciplining children, marijuana legality, universal basic income, or immigration split people along cultural, political, socioeconomic, gender, and religious lines. A response to any of these will upset some users, while a model that avoids them entirely may become boring and lose users. Keeping this balance is the core challenge that preference finetuning is meant to handle.

RLHF, short for Reinforcement Learning from Human Feedback, is the earliest successful preference finetuning algorithm and is still popular today. It has two parts. The first part trains a reward model that scores the foundation model's outputs. The second part optimizes the foundation model so that it generates responses the reward model scores as highly as possible.

The diagram should be read as a loop. The foundation model produces outputs, the reward model assigns each output a score, and the optimization step uses those scores to change the foundation model so that future outputs earn higher scores. The two parts are not independent. The reward model defines what counts as a good response, and the optimization step is how the foundation model learns to produce those responses. If the reward model is poor, the optimization step faithfully pushes the model toward the wrong target.

A common first idea is to collect many responses to prompts, ask people to rate each one on a numeric scale, and train a model to predict those ratings. The resulting model would assign a number to any prompt and response pair. This idea sounds complete, and the reward model is meant to output exactly such a score. The trouble appears when you look closely at what labelers actually produce.

Asking a labeler to score each response independently is called pointwise evaluation. Each sample is judged alone, against no other response. The source's illustration uses a 10-point scale: for the same sample, one labeler might give a 5 and another a 7. The definitions in the source also note that scores vary across labelers and even within one labeler over time. Since each person is calibrating their own meaning of a number, absolute scores are a weak foundation for training.

The alternative is to ask labelers a simpler question: given two responses, which one is better? For each prompt, multiple responses are generated, either by humans or by AI. Labelers compare them, and the result is called comparison data. Each record follows the format (prompt, winning_response, losing_response). The record stores the prompt, the response judged better, and the response judged worse.

Comparison data is easier to obtain because labelers do not have to agree on what a numeric value means. They only have to say which of two responses is better. The source also cautions that a labeler may still disagree with what was recorded, a point that returns when we discuss labeler disagreement later in this article.

Comparison data gives preferences, but the reward model must output concrete scores, so the question is how a model learns numbers from pairwise judgments. The source's answer is an objective function. A commonly used function represents the difference between the output scores for the winning and losing response, and training maximizes that difference. The idea is that a good reward model should give the winning response a clearly higher score than the losing one.

The source describes this objective as a log-sigmoid objective. For each training sample, the score gap between the winning and losing response is passed through the sigmoid and then the logarithm, and training seeks the parameters that minimize the expected loss across all samples. A larger gap between the winning score and the losing score makes the term larger, which is why the objective rewards separating the two scores in the correct direction.

The reward model takes a prompt and a response and produces a single number for how good the response is. The source calls this a scalar score. Its value is useful mainly in relation to other scores the model produces. Because training is built on differences between winners and losers, the scale is learned relative to the comparisons. It is not fixed by any human's private numeric meaning, which is exactly what made direct ratings unreliable.

RLHF is not the only approach. DPO, which stands for Direct Preference Optimization, is a newer alternative that is gaining traction and reduces complexity. The source's example is Meta, which moved from RLHF for Llama 2 to DPO for Llama 3 in order to reduce complexity. The trade-off is that RLHF remains in use because it offers more flexibility to tweak the model, while DPO gives up some of that flexibility in exchange for skipping the separate reward model.

Comparison data is cheaper and more consistent than direct scores, but it still comes from people. The source reports around 73 percent inter-labeler agreement for OpenAI's labelers. In practical terms, if 10 people rank the same two responses, about 7 would give the same ranking and 3 would disagree. The label is therefore a strong majority signal, not a certainty, and the noise from the remaining disagreement flows into the reward model that learns from those labels.

Cost is the second constraint. The source reports that Scialom shared a cost of 3.50 dollars per comparison, while writing a response costs 25 dollars. LMSYS found that manually comparing two responses took on average three to five minutes, because the process requires fact-checking each response. These figures explain why comparison data is attractive, and also why the amount of it is limited in practice.

The reward model can be trained from scratch, or it can be finetuned on top of another model such as a pre-trained model or an SFT model. The source states that finetuning on top of the strongest foundation model seems to give the best performance. The practical lesson is that the starting point matters: a stronger base gives the training process a better foundation to adjust.