Pre-Training Foundation Models

Building a language-based foundation model usually happens in stages. The first stage is pre-training, which uses self-supervision to produce a model that is capable in a broad sense. That model is a raw starting point, not a finished product. It is not yet ready to act as a helpful conversational partner. Post-training is the set of steps that takes this raw model and reshapes it so that it behaves the way people actually want it to behave.

A common assumption is that post-training is where a model learns almost everything it knows, and that pre-training only lays a rough foundation. The reasoning is that post-training is the step that makes the model useful, so it must supply most of its ability. This belief is understandable, but it describes the relationship backwards. Pre-training is the expensive stage that builds the model's broad knowledge. Post-training unlocks capabilities the pre-trained model already has, capabilities that are hard for users to reach through prompting alone.

Consider what a pre-trained model does with a prompt. Its training objective is text completion: it has learned to continue a piece of text in a plausible way. Given the start of a passage, it produces more text that fits. A model trained this way is good at continuing text, but continuing text is not the same task as holding a helpful exchange with a person. A plausible continuation of a question may be more questions, or a paragraph in the same style, rather than an answer.

The two problems are different in kind. One is about the training objective, meaning what the model is rewarded for during training. The other is about the data the model learns from. Treating them separately makes it easier to see which post-training step addresses which problem.

Pre-training and post-training optimize different things. For language-based foundation models, pre-training optimizes token-level quality. The model is trained to predict the next token accurately. A token is a small piece of text, so getting each next piece right is a local measure of quality.

Users do not judge a model one token at a time. They care about the quality of the entire response. A reply can be built from individually plausible pieces and still fail as an answer. Post-training therefore optimizes the model to generate whole responses that users prefer. The target moves from the small unit, the token, to the whole reply.

In general, post-training consists of two steps, applied in a fixed order. The first is supervised finetuning, usually abbreviated SFT. The second is preference finetuning. The order matters because preference finetuning builds on a model that has already been supervised finetuned.

The pre-trained model passes through supervised finetuning first, and then through preference finetuning.

Supervised finetuning trains the model on higher-quality data. This addresses the data-quality problem: the model is pushed toward the kind of text that is more appropriate than what indiscriminate scraping provided. Preference finetuning then shapes the model toward responses that suit customers, which addresses the goal of producing replies people prefer rather than merely plausible text.

The full workflow includes pre-training, supervised finetuning, and preference finetuning. The source describes a combination of these as the popular approach for building foundation models today. It is not the only approach, and some steps can be skipped. Because of this, you can approximate how well a model aligns with human preference by checking which of these steps its creators took.

A helpful picture for these stages is a monster and its training. Self-supervised pre-training produces a rogue model, an untamed monster, because it learns from indiscriminate internet data. Supervised finetuning on higher-quality data tames the monster and makes it more socially acceptable. Preference finetuning then polishes the tamed model so it suits customers, which the picture describes as giving it a smiley face.

Several related terms appear in discussions of post-training, and they do not always mean the same thing. Post-training is the broad stage that starts from a pre-trained model. Supervised finetuning is one of the two steps inside that stage. Preference finetuning is the other step. The term instruction finetuning is used loosely, and a reader can encounter it meaning different things in different materials. Some uses point to the whole post-training stage, while others point to the supervised step that trains on higher-quality data.