Post-Training: Supervised and Preference Finetuning
Large models need large amounts of training data, which is expensive and slow to collect, so providers often use whatever data is available, and that may leave gaps for specific tasks.
A Hospital Assistant That Must Read Clinical Notes
Imagine a hospital network that wants an assistant able to read and summarize clinical notes. Some of those notes are written in a regional language, and others use one specialized medical vocabulary. Before anyone chooses a model design, a more basic question comes up: what has the model seen? A model can only become good at patterns that appear in the material it learned from. For foundation models, that material is the training data, and it is one of the most important factors in how a model performs.
This article follows the hospital scenario through the rest of the design questions: the training data, the size of the model, the way the model is refined after its first training phase, and the way it produces output. Each of these choices connects back to the data question, so keep the hospital assistant in mind as you read.
Why Training Data Sets the Limits
Large models need large amounts of training data. Collecting that data is expensive and slow, so providers often use whatever data is available. That choice can leave gaps for specific tasks.
Quantity matters, but so does quality and coverage. A model can only learn patterns that appear in its data. If a language or a specialized domain is underrepresented, the model has little to learn from in that area. The source notes that curating training data is often necessary for specific low-resource languages and specialized domains.
Return to the hospital assistant. A general-purpose model trained mostly on common languages and everyday text may handle a standard English discharge summary reasonably well. The same model may struggle with a note in a regional language or with dense specialized medical vocabulary, because those patterns were scarce or absent in what it learned from. The gap is not a bug in the model's reasoning. It reflects what the training data covered.
Measuring Model Scale
Model scale is described with three numbers. Parameters count the size of the model. Training tokens count the amount of data used to train it. Training compute is the work needed to train the model, measured in FLOPs. The three numbers are not independent. The size of the model and the size of the dataset are the two main factors that determine how much compute training requires, so a bigger model or a larger dataset generally means more training work.
- Parameters: the size of the model
- Training tokens: the size of the dataset used for training
- Training compute (FLOPs): the work required to train the model
- The three must be read together, because model size and dataset size both drive the compute requirement
Reading the Compute Budget Through a Scaling Law
When compute is limited, a designer has to decide how to spend it. Model size and data size both draw on the same compute budget, so increasing one leaves less room for the other. A scaling law is the rule that reads the budget and suggests a balanced allocation between parameters and training tokens. It helps find the best balance for a given compute budget.
Read the diagram from the top. The compute budget is the fixed total. Parameters and training tokens both come out of it, so spending more on one leaves less for the other. The scaling law sits beside the budget and reads it, then points toward a balanced allocation between the two.
Walking Through a Fixed Budget
Shifting a Fixed Compute Budget
A team has one fixed compute budget for training. They are deciding whether to use a larger model trained on fewer tokens, or a smaller model trained on more tokens. What does the scaling law tell them to consider?
- 1Identify the fixed quantity
The total compute budget is fixed. That means the total training work cannot grow, whatever the team chooses.
- 2Identify what competes for it
Model size (parameters) and dataset size (training tokens) both draw on the same budget. Each is a share of the same total.
- 3Apply the trade-off
If the team puts more of the budget into parameters, less remains for training tokens. If it puts more into tokens, less remains for parameters.
- 4Consult the scaling law
The scaling law reads the budget and suggests a balanced allocation between parameters and tokens. The team should look for the split the law recommends, not pick either extreme by default.
The choice is a balance, not a free decision. A larger model is not automatically better if it leaves too little data for training within the same budget. The scaling law is the tool for finding that balance.
Post-Training After Pretraining
Pretraining gives a model its broad base of learned patterns. But a pretrained model's outputs do not always line up with what users want. Post-training is the stage that addresses this. It has two steps: supervised finetuning first, then preference finetuning. Together they target outputs that do not align with what users want.
Follow the model through the pipeline. It starts as a pretrained model, shaped by its training data. Supervised finetuning is applied next, and preference finetuning comes after it. The final model is refined, but the process has a limit. Human preference is diverse and cannot be captured in a single mathematical formula, so the methods available for this are far from foolproof. Post-training narrows the gap between what a pretrained model produces and what people want, but it does not guarantee a perfect match.
Why Outputs Are Sampled
A model does not always produce the same output for the same prompt. Its outputs are produced by sampling, which makes them probabilistic. Sampling brings both benefits and risks. The benefit is that the model can produce varied responses rather than one fixed answer each time. The risk is that the output is not fully predictable, so the same prompt can lead to different results on different runs.
Picture the hospital assistant summarizing the same clinical note twice. Because the output is sampled, the two summaries may differ in wording or emphasis. That variety can be useful when drafting, but it also means a reviewer cannot assume a single run reflects every possible output.
Mistakes Beginners Make About Training and Post-Training
Common mistake Assuming a larger dataset always covers every task
What it looks likeA team expects a model to handle a regional-language clinical note because the training set is very large.
Why it failsData quantity does not guarantee coverage. If a language or specialized vocabulary is barely present, the model has little to learn from there.
Do this insteadCheck whether the data covers the specific language and domain. Curating data is often necessary for low-resource languages and specialized domains.
Common mistake Treating parameters as the only measure of scale
What it looks likeA team compares two models only by parameter count and ignores how much training data each one used.
Why it failsParameters, training tokens, and training compute must be read together. Model size and dataset size both drive the compute needed.
Do this insteadCompare all three measures, and consider how the compute budget was split between parameters and tokens.
Common mistake Believing a bigger model is always the better choice under a fixed budget
What it looks likeA team picks the largest model it can afford without considering how little training data remains.
Why it failsWithin a fixed compute budget, model size and data size compete. Spending everything on parameters leaves less room for training tokens.
Do this insteadUse a scaling law to find a balanced allocation between parameters and tokens.
Common mistake Assuming post-training makes outputs perfectly match user wants
Why it failsPost-training narrows the gap but does not guarantee a perfect match, because human preference is diverse and hard to capture in a single formula.
Do this insteadTreat post-training as an improvement that still needs evaluation for the specific use.
Common mistake Expecting identical outputs from identical prompts without considering that outputs are sampled
Check Your Reasoning on the Hospital Scenario
The hospital team has a fixed compute budget and a model that handles standard English notes well but struggles with notes in a regional language. Using only what this article covered, explain two separate reasons the model might struggle, and say which post-training step addresses output alignment rather than data coverage.
Hints
- Think about what the training data contained, which is a data coverage question.
- Think about the fixed budget: parameters and tokens compete for it.
- Recall the two post-training steps: supervised finetuning and preference finetuning.
Key Takeaways
- Training data sets the limits of what a foundation model can do. Data is expensive to collect, so gaps can appear for low-resource languages and specialized domains.
- Model scale is described by parameters, training tokens, and training compute (FLOPs). These must be read together because model size and dataset size both drive training work.
- Model size and data size draw on the same fixed compute budget. A scaling law reads that budget and suggests a balanced allocation between parameters and tokens.
- Post-training follows pretraining and has two steps, supervised finetuning and preference finetuning, which address outputs that do not align with what users want. It narrows the gap but does not guarantee a perfect match.
- Outputs are produced by sampling, so they are probabilistic. This allows varied responses but also means outputs are not fully predictable.