Pre-training Large Language Models

Suppose someone tells you they have just trained a language model and asks what that means. Before you can answer, you need a shared idea of what training is, and you need to decide whether every operation that touches a model's weights belongs in that category. This article builds that idea. It separates the general word training from the more specific words pre-training, fine-tuning, and post-training, and it asks where other operations, such as quantization, fit.

The clearest way to see the relationship between the phases is to ask where each one starts. Pre-training starts from randomly initialized weights. Fine-tuning starts from a model that has already been pre-trained. Post-training comes after fine-tuning and starts from the fine-tuned model. The phases therefore form a chain, and each stage begins from the result of the stage before it.

Read the diagram from left to right. Each phase depends on the phase before it. Pre-training does the first, foundational work. Fine-tuning adjusts an already pre-trained model toward one task. Post-training then improves the model further after fine-tuning. Because each stage begins from a model that already carries the results of earlier work, you cannot understand a later phase without knowing what came before it.

The definition has two halves, and both matter. The first half says that training changes weights. Changing weights is necessary for training. The second half says that a weight change is not automatically training. The purpose of the change is what makes it training: the change is made to improve performance. This is the reason the source uses quantization as a counterexample. Quantization alters the stored weight values, but it is treated as a separate operation rather than a phase of training.

Quantization is the process of reducing the precision of model weights. It is worth studying next to training because it touches the same objects, the weights, and it can change their values. The difference is what happens to the model. When quantization occurs, the stored value becomes less precise, and the model is not being trained to get better at a task. The contrast is what makes the distinction useful.

Pre-training starts from randomly initialized weights. For large language models, the source says pre-training often involves training the model for text completion. The model is trained to continue text. This general capability is the foundation that later phases build on. Since pre-training begins from random weights, every later phase starts from a model that already has the results of this work.

The word training in everyday speech often serves as a shorthand that covers the more specific phases. The source notes that the term training can often be used in place of pre-training, fine-tuning, and post-training. When you hear that someone trained a model, the phrase alone does not tell you which phase they mean. Asking where the process started, whether from random weights, a pre-trained model, or a fine-tuned model, is the quickest way to find out.

Of all the training steps, pre-training is often the most resource-intensive by a long way. The source gives a concrete figure: for the InstructGPT model, pre-training takes up to 98 percent of the overall compute and data resources. The remaining phases together use a small share of what the project spends.

The same resource demands raise the stakes. Pre-training takes a long time, and a small mistake during it can cause a significant financial loss and set the project back considerably. The source observes that because of this, pre-training has become a skill that only a few people practice, and those with expertise in pre-training large models are heavily sought after.