Reinforcement Learning for Language Models
SFT is the step that moves the model toward conversational behavior. It is the first post-training step, and it uses high-quality instruction data as its training material.
Where Post-Training Fits
A pre-trained language model has already learned from a large amount of data. That learning is not the end of the story. Post-training is the set of steps that comes after pre-training, and its job is to refine the model so that it responds well in a conversation. As you read, keep one question in mind: what is the model being optimized for at each step, and how does that target differ from the one before it?
The source names two steps in post-training: supervised finetuning and preference finetuning. The first moves the model toward conversational behavior, and the second moves it toward responses that people prefer. The diagram below shows the order. The pre-trained model is the starting point, and each arrow is one finetuning step that changes the model's target.
Supervised Finetuning and Conversational Behavior
Supervised finetuning (SFT) finetunes the pre-trained model on high-quality instruction data to optimize it for conversations instead of completion.
The key word in that definition is target. A pre-trained model is shaped to continue text. SFT changes the goal: the model is trained on instruction data, so it learns to respond to a request the way a helpful conversational partner would. The purpose of SFT, then, is to change what kind of output the model is aiming for, from continuing text to answering in a conversation.
SFT is the step that moves the model toward conversational behavior. It is the first post-training step, and it uses high-quality instruction data as its training material.
Preference Finetuning on Top of SFT
Preference finetuning further finetunes the model to output responses that align with human preference, and it is typically done with reinforcement learning.
Preference finetuning comes after SFT, not instead of it. By the time it begins, the model already behaves conversationally. The new target is narrower and more specific: among the responses the model could give, it should favor the ones people prefer. The source describes this as further finetuning, which means the model's existing behavior is adjusted rather than rebuilt from scratch.
Why Reinforcement Learning Is Used
SFT teaches the model by showing it fixed target answers and asking it to imitate them. Preference finetuning has a different target: responses that people prefer. The source states that this step is typically done with reinforcement learning. The comparison below contrasts the two approaches in terms of the data each uses and how the model learns from it.
The loop below shows how feedback flows in this kind of training. The model produces responses, preferences score those responses, and the model's weights are updated based on that feedback. The same cycle then repeats, which is why the process is described as a loop rather than a single pass over fixed data. This is a general description of the RL idea rather than a detail the source spells out step by step.
Tracing One Model Through the Steps
To see what changes, follow one model through the sequence. The scenario below is an illustrative example created for this article. It is not a result reported in the source, but it shows how the model's goal shifts at each stage.
One Prompt Across Three Stages
A user asks a model: Can you explain why the sky looks blue? Trace how the model's target changes from the pre-trained stage through SFT to preference finetuning.
- 1Stage 1: pre-trained model
The model has learned from a large amount of data and is shaped to continue text. Its target is completion, so it may simply extend the prompt with more text that looks like it belongs nearby.
- 2Stage 2: after SFT
The model has been finetuned on high-quality instruction data. Its target is now conversational, so it responds to the request as an answer to a user.
- 3Stage 3: after preference finetuning
The model has been further finetuned toward responses that align with human preference. Among the answers it could give, it now favors the ones people prefer, so the target is narrower than in Stage 2.
Each stage keeps the previous stage's work and changes the target. SFT changes the model from completing text to conversing. Preference finetuning changes it from conversing to conversing in ways people prefer.
Mistakes Beginners Make
Common mistake Treating SFT and preference finetuning as the same step
What it looks likeSaying that SFT already makes the model prefer responses people like.
Why it failsThe source describes SFT as optimizing the model for conversations, and preference finetuning as a separate, further step that aligns responses with human preference.
Do this insteadKeep two distinct targets in mind: conversational behavior for SFT, and preferred responses for preference finetuning.
Common mistake Reversing the order of the post-training steps
What it looks likeRunning preference finetuning directly on the pre-trained model and then adding SFT.
Why it failsThe source states the order: pre-trained model, then SFT, then preference finetuning.
Do this insteadAlways place SFT between the pre-trained model and preference finetuning.
Common mistake Assuming preference finetuning replaces pre-training
Why it failsPost-training is the set of steps that comes after pre-training and refines the model. It does not redo the pre-training stage.
Do this insteadThink of post-training as refinement of an existing model, not as a new starting point.
Common mistake Describing RL as simply more of the same supervised imitation, when the source ties preference finetuning to reinforcement learning rather than to fixed target answers alone
Check Your Understanding
A team has a pre-trained model. They want it to answer user requests in a conversation, and then they want its responses to match what people prefer. Name the two post-training steps in the order they should run, and state what each step changes about the model.
Hints
- Look for which step the source ties to conversational behavior and which to human preference.
- The source states the order: pre-trained model, then SFT, then preference finetuning.
Summary of Post-Training
- 1Post-training refines a pre-trained model after pre-training so that it behaves well in a conversation.
- 2Supervised finetuning finetunes the model on high-quality instruction data to optimize it for conversations instead of completion.
- 3Preference finetuning further finetunes the model toward responses that align with human preference, and it is typically done with reinforcement learning.
- 4The steps run in order: pre-trained model, then SFT, then preference finetuning, and each step changes the model's target.
At a glance
- Post-training refines a pre-trained model after pre-training so that it responds well in a conversation.
- SFT finetunes the pre-trained model on high-quality instruction data to optimize it for conversations instead of completion.
- Preference finetuning further finetunes the model to output responses that align with human preference, and it is typically done with reinforcement learning.
- The order is pre-trained model, then SFT, then preference finetuning, and each step changes the model's target.