Reward Model Training
The reward model scores the responses that the SFT model generates for sampled prompts, and it does not write the responses itself.
Why a Judge Needs Something to Judge
Once a reward model has been trained, a natural question follows: how do we use it? A reward model does not write answers to user questions. Its job is to judge answers that some other model has written. This article follows the step where that judgment is fed back into the model that produced the answers, so that the model learns to produce answers the judge rates more highly.
Keep this idea in view as you read: a judge cannot improve a model on its own. A score by itself is only a number. It becomes useful when something uses that number to change what the generating model does next time, or when something uses it to choose among outputs the model has already produced. Both of those paths are covered here.
Two Models, Two Jobs
There are two models in play. The SFT model generates responses. The reward model scores them. These roles never swap. The SFT model is the one that writes text in reply to a prompt, and the reward model is the one that assigns a score to that text.
The finetuning step changes the SFT model. The reward model is the reference point that tells the SFT model whether a change was an improvement. In this stage the reward model acts as a fixed judge: it is consulted for every response, but it is not the thing being trained.
| Component | What it receives | What it outputs | Is it changed in this stage? |
|---|---|---|---|
| SFT model | A prompt | A response written as text | Yes, its parameters are adjusted |
| Reward model | A prompt and a candidate response | A numeric score | No, it serves as a fixed judge |
The division of roles between the two models during finetuning
Following a Prompt Through the Scoring Loop
Training begins with prompts. They are selected at random from a distribution of prompts, and existing user prompts are one example of where such prompts can come from. Trace one prompt through the system, and watch where the information goes at each step.
The score is the output of the whole chain. Prompts go in, responses come out of the SFT model, and the reward model turns each response into a score. Notice where the score ends up: it does not go back to the user. It goes into the training process, where it shapes the next version of the SFT model.
How PPO Pushes the SFT Model Toward Higher Scores
The stated goal of this stage is that the SFT model is further trained to generate output responses that maximize the scores given by the reward model. The score is the signal, and the SFT model's parameters are what get adjusted in response to it.
The source names the method usually used for this training: proximal policy optimization, abbreviated PPO. It is a reinforcement learning algorithm, released by OpenAI in 2017. In this setting, PPO is the mechanism that takes the reward model's scores and uses them to train the SFT model so that its responses earn higher scores.
Read this diagram as a loop rather than a single pass. The reward model is consulted for every response, and the PPO update is the step that alters the SFT model. Because prompts are sampled repeatedly, the same cycle happens across many different inputs. Each round starts from the model as it was after the previous update, which is why the process is described as a cycle.
Tracing Four Candidate Responses
Scoring four responses to one prompt
The SFT model is asked one prompt and produces four different candidate responses, labeled A through D. The reward model gives each candidate a score. Which candidate should the finetuning signal favor, and what does the score contribute?
- 1Gather the candidates
The same prompt was sent to the SFT model, which produced four different responses. Because of sampling, the model did not have to write the same text each time.
- 2Pass each candidate to the reward model
Each response is handed to the reward model separately. The reward model does not write anything. It only assigns a score to each response it receives.
- 3Read the scores
Illustrative scores for this example (generated for teaching, not from the source): A 0.42, B 0.87, C 0.15, D 0.61. Higher means the reward model rated the response more highly.
- 4Identify the direction of the signal
PPO trains the SFT model to produce responses that maximize reward scores, so the signal points toward the behavior that produced response B, the highest score here.
Response B has the highest score, 0.87, so the training signal favors the kind of response that B represents. The scores themselves are the only information the reward model contributes; the reward model did not produce any of the four responses.
What do you think happens?
Before reading on: using only the four scores A 0.42, B 0.87, C 0.15, D 0.61, which response should the finetuning signal favor?
Reveal answer
Response B, the one with the highest score of 0.87.
PPO trains the SFT model to produce responses that maximize the reward model's scores, so the highest-scoring response is the one the signal favors. Lower scores such as C at 0.15 push the model away from that kind of output.
Best of N: Selecting Instead of Retraining
Reinforcement learning is not the only way a reward model can help. The source describes companies that skip reinforcement learning altogether. Stitch Fix and Grab, for example, find that having the reward model alone is good enough for their applications. Instead of updating the model, they have it generate multiple outputs and keep the ones the reward model scores highly.
This approach is often referred to as the best of N strategy. It leverages how a model samples outputs to improve its performance. The mechanism is selection: the model is not changed, but the output the user receives is chosen from several candidates by the reward model.
The pipeline shows the difference from PPO. Nothing in the SFT model's weights is required to change. Many candidates are generated, each is scored, and only the top-scoring one moves forward. The quality gain comes from the selection step rather than from retraining.
| Question | PPO | Best of N |
|---|---|---|
| Are the model's parameters changed? | Yes, the SFT model is trained | No, the model is not changed |
| Is reinforcement learning used? | Yes, PPO is a reinforcement learning algorithm | No, the reward model alone selects the output |
| Where does the quality gain come from? | Training the model toward higher scores | Choosing the best of several sampled candidates |
| What does the reward model do? | Supplies the scores that drive training | Scores candidates so the top one is chosen |
How PPO and best of N each raise reward scores
Why Sampling Makes Best of N Possible
Best of N only works because a model does not produce one fixed answer. The source explains that a model constructs its outputs through a process known as sampling. Sampling makes the outputs probabilistic, meaning that the same prompt can lead to different responses on different runs.
That variability is what gives the reward model something to choose from. If every run produced the identical response, generating N outputs would produce N copies of one answer, and selection would have nothing to improve. Sampling is therefore the source of the diversity that the best of N strategy depends on.
Mistakes Learners Make About Reward Models
Common mistake Thinking the reward model writes the answers
What it looks likeAssuming that the reward model is the chatbot the user talks to.
Why it failsThe source states that the reward model does not write responses. It judges answers that another model, the SFT model, has written.
Do this insteadKeep the roles separate: the SFT model generates text, and the reward model returns a score.
Common mistake Believing the reward model is retrained during PPO
What it looks likeThinking both models are updated in every round of the loop.
Why it failsThe reward model is the fixed reference point. The finetuning step changes the SFT model.
Do this insteadDuring PPO, only the SFT model's parameters are adjusted by the score signal.
Common mistake Assuming best of N must involve reinforcement learning
What it looks likeThinking that any method using a reward model is a form of PPO.
Why it failsThe source describes companies that use the reward model alone to pick the best sampled output, with no reinforcement learning and no changes to the model.
Do this insteadBest of N generates several outputs, scores them, and keeps the top one.
Common mistake Treating a single sampled response as the model's true answer, when a different run of the same prompt could produce a different response
Practice: Choose the Right Strategy
A team has a working SFT model and a trained reward model. They want better responses but do not want to run reinforcement learning. Which strategy fits, and why does the model's weights stay the same under it? Explain your answer in terms of sampling and the reward model's role.
Hints
- Ask whether the team wants to change the model's parameters or only choose among outputs.
- Think about what makes multiple outputs for the same prompt possible.
- Remember that the reward model scores outputs and does not write them.
Key Takeaways
- A reward model does not write responses. It scores the responses the SFT model generates for sampled prompts, and it acts as a fixed judge during finetuning.
- PPO, a reinforcement learning algorithm released by OpenAI in 2017, uses the reward model's scores to adjust the SFT model so its responses earn higher scores, in a repeated cycle over many prompts.
- The best of N strategy samples several outputs from the same prompt, scores each with the reward model, and returns the highest-scoring one without changing the model or using reinforcement learning.
- Sampling makes outputs probabilistic, which gives best of N its candidates, but the same behavior also contributes to inconsistency and hallucination.