Pretraining a Large Language Model

Imagine you want a program that can write fluent sentences. You could try writing rules for grammar and word choice by hand, but language is too varied for that to work. A different approach is to let a model learn patterns from a large body of text and then use what it learned to generate new text. The process of building that learned starting point is called pretraining. It is the stage where a model is first trained on text before anyone asks it to do a specific job.

The process that closes this gap is instruction fine-tuning. This article focuses on the process itself: what goes into it, what happens during it, and what the model looks like afterward. Throughout, keep one idea in mind. The model is not built from scratch in this step. It starts from a pretrained LLM and is adjusted so that its behavior shifts toward following instructions.

The key word in this definition is adapting. The definition names two outcomes: the model follows human instructions, and it generates the responses that were desired. Both outcomes are about behavior. The model is not given new facts from scratch; its existing behavior is shifted in a particular direction.

The easiest way to picture instruction fine-tuning is to follow the model from its starting state to its finished state. The model's state is its set of learned weights. Before fine-tuning, the weights hold what pretraining produced. During fine-tuning, the training loop runs on instruction data and nudges those weights. After fine-tuning, the weights are the changed state, and the model is the one that follows human instructions.

Read the diagram from left to right. The pretrained model and the fine-tuned model are the same kind of object at different points in the process. The difference is the state each one is in. The only node marked as changed at the end is the fine-tuned LLM, because its behavior has been shifted toward following human instructions.

Instruction fine-tuning combines three ingredients: a pretrained model, an instruction dataset, and a training loop. Each one plays a distinct role, and the process cannot be understood without all three. The sections below describe each ingredient in turn.

The source states that the pretrained model is fine-tuned on the instruction dataset using a training loop similar to the one used in pretraining. This means the mechanics are familiar. The model is run through training iterations, just as it was during pretraining. The loop repeats a sequence of steps: a forward pass, a loss computation, a backward pass, and a weight update. Each pass through these steps adjusts the weights a little.

The steps above are the same in both stages. What differs is the data. Pretraining was the first training stage, and it learned from text. Instruction fine-tuning continues training on instruction data, with the aim of changing the model's behavior toward following instructions. So the procedure stays the same while the data and the goal change.

In this course, pretraining is treated as the second major stage of work. The first stage prepares the ingredients: data preprocessing and the attention mechanism at the core of modern LLMs. The second stage puts those ingredients together, showing how to code and pretrain a GPT-like model that can generate new text, and then how to evaluate the result. Keeping these stages in mind helps you see why each topic exists.

Read the diagram from left to right. The first two steps belong to stage 1 and concern preparing data and understanding attention. The pretraining step belongs to stage 2. The branch for loading open model weights sits beside pretraining because it is an alternative way to reach a trained model without running the full training yourself. Both routes end at evaluation.

Data preprocessing is a crucial step in pretraining an LLM. Raw text is rarely in the shape a model can learn from directly, so it has to be prepared first. Preprocessing is the part of the workflow that turns collected text into training material. Because every later step depends on this preparation, it is worth understanding before you move on to the model itself.

The attention mechanism is the second ingredient from stage 1. The source describes it as being at the heart of every LLM, and it is the component the course asks you to code yourself. Think of attention as the part of the model that lets each position in a sequence take other positions into account when it builds its representation. Stage 1 builds this mechanism in isolation, so that stage 2 can use it as a working part rather than a mystery.

There are two routes to a trained model. You can pretrain one yourself, or you can load openly available model weights that someone else produced. Pretraining an LLM from scratch is a significant undertaking. The source states that training a GPT-like model from scratch costs thousands to millions of dollars. That range is wide, so the practical lesson is that full-scale pretraining is a major commitment rather than a routine exercise.

This cost is the reason the course narrows its goal. Instead of pretraining at full scale, stage 2 implements training for educational purposes using a small dataset. The purpose is to understand how the process works, not to reproduce a large production model.

The third topic in stage 2 is evaluating LLMs. The source describes this as essential for developing capable NLP systems. A trained model is a set of learned values, and those values alone do not tell you whether the model produces good text. Evaluation is the step that measures what the model does, so that you can compare versions and decide what to change.

Evaluation applies to both routes. A model you pretrain on a small dataset needs to be evaluated, and so does a model whose weights you loaded from an open source. Because the weights in the second case came from elsewhere, you cannot assume they behave as you expect in your own setting. Measuring is how you find out.