Sampling in AI Models
Post-training refines a pre-trained model so that its outputs better align with human preferences and task-specific requirements.
A Model That Can Talk but Still Misses
Imagine a large language model that has finished its initial training. It can continue a piece of text, answer questions, and produce fluent sentences. Yet its replies may still feel off. They can be too rambling, they can seem indifferent to what the person actually asked for, or they can simply be the kind of answer a user would not prefer. The gap is between what the model learned during its initial training and what people actually want from it.
Post-training is the set of techniques built to close that gap. Before working through those techniques, it helps to predict what might happen next. Suppose a model is already capable of fluent text. Would you expect fluency alone to make its answers useful to every user? Keep that question in mind as you read the next sections.
What do you think happens?
A pre-trained model is fluent and can answer questions. Does that guarantee its answers match what users want?
Reveal answer
No, its outputs may still not align with what users want
The source describes a pre-trained model that can continue text, answer questions, and produce fluent sentences, yet still give responses that feel off, too rambling or indifferent to what the person wanted. Post-training exists to address this mismatch.
Where the Mismatch Comes From
The reason post-training is needed traces back to how a model learns in the first place. The training data a pre-trained model learns from is often of low quality, and pre-training relies on self-supervision. Together, these factors can leave a model whose outputs do not line up with what users want. Post-training is the stage that addresses this mismatch.
A useful informal picture comes from a comment made about this stage of development: a pre-trained model talks like a web page, not like a human. A web page can contain a great deal of information, but it does not adjust itself to the reader. Post-training is the work of shaping that raw language ability into a conversational partner that responds in ways people find useful.
Post-training is not the first stage of building a model. It comes after pre-training and works on the existing model rather than replacing it. The pre-trained model is the starting material, and each post-training step reshapes it further before it reaches users.
The Two Post-Training Steps
Post-training consists of two steps applied in a fixed order. The first is supervised finetuning, which finetunes the model on a specific task or dataset. The second is preference finetuning, which aligns the model with human preferences. Each step addresses a different part of the alignment problem.
Read the diagram from left to right. The pre-trained model enters supervised finetuning, and that step's output enters preference finetuning. What comes out at the end is the post-trained model. The order matters, because the second step is applied to a model that has already been through the first.
Sampling and Probabilistic Output
Post-training connects to a broader property of AI models. Sampling is the process by which a model generates output tokens, and it makes models probabilistic. Because the output is produced by sampling rather than a single fixed rule, the same prompt does not always yield the same response.
That probabilistic nature is part of what makes models like ChatGPT and Gemini good for creative tasks and pleasant to talk with. It also causes inconsistency and hallucinations. Post-training is one of the tools used to bring model behavior closer to what people expect, while the probabilistic character of sampling remains part of how these systems work.
Before and After Post-Training
The effect of post-training can be seen by comparing the same prompt answered by the pre-trained model and by the post-trained model. The source does not provide sample outputs, so the comparison below is an illustrative generated example of the kind of change the source describes: a reply that is off-target versus one that fits the user.
Prompt: What is a good way to start learning Python? Pre-trained reply: a long, wandering passage that touches many topics and never settles on a clear recommendation.
Pre-trained model: fluent but indifferent to what the user wanted
Prompt: What is a good way to start learning Python? Post-trained reply: a short, direct recommendation that answers the question the user asked.
Post-trained model: shaped toward a useful conversational answer
Why Full Alignment Is Hard
Preference finetuning cannot guarantee full alignment. Human preference is diverse, and it is impossible to capture in a single mathematical formula. Different people want different things from the same answer, so a single target to optimize toward is always a simplification. Post-training therefore improves alignment without guaranteeing it.
Alignment is also not always the goal in every setting. The source gives an example in which a misaligned model might be useful. If you want to evaluate the risk that people could use AI to spread misinformation, you might deliberately build a model that is as good as possible at producing convincing fake news, so you can see how persuasive AI can be. In that case the aim is to study a model's misbehavior, the opposite of what post-training normally targets.
Mistakes Learners Make About Post-Training
Common mistake Assuming a pre-trained model is already aligned with user wants
Why it failsThe source says a pre-trained model may produce outputs that do not align with what users want, because of low-quality training data and self-supervision during pretraining.
Do this insteadTreat alignment as a separate job that post-training performs after pre-training.
Common mistake Reversing the order of the two post-training steps
Why it failsPost-training consists of two steps applied in order, and the second step is applied to a model that has already been through the first.
Do this insteadApply supervised finetuning first, then preference finetuning.
Common mistake Believing post-training guarantees alignment
Why it failsHuman preference is diverse and cannot be captured in a single mathematical formula, so post-training improves alignment without guaranteeing it.
Do this insteadDescribe post-training as improving alignment, not perfecting it.
Common mistake Treating sampling as if it always returns the same output for the same prompt
Checking Your Understanding of the Pipeline
Suppose a team swaps the order so that preference finetuning runs first and supervised finetuning runs second. Using the source's description of the pipeline, explain what goes wrong or why the order matters, and name the purpose of each step.
Hints
- Recall that the second step is applied to a model that has already been through the first.
- Supervised finetuning targets a specific task or dataset; preference finetuning targets human preferences.
Key Takeaways
- A pre-trained model can be fluent yet still give outputs that do not align with what users want, partly because of low-quality training data and self-supervision during pretraining.
- Post-training has two steps applied in order: supervised finetuning on a specific task or dataset, then preference finetuning to align the model with human preferences.
- Sampling generates output tokens and makes models probabilistic, which supports creativity but also causes inconsistency and hallucinations.
- Human preference is diverse and cannot be captured in a single formula, so post-training improves alignment without fully guaranteeing it.