Pretraining Large Language Models

Imagine a model that has read a very large amount of varied text. It has absorbed a broad sense of how language works, but it was not built for any one job. You might want it to sort your inbox into spam and not spam, or to follow instructions and answer in a particular format. A model that only learned general language does not automatically do these specific things well. This article is about the step that bridges that gap, called fine-tuning, and about the stage that comes before it.

Keep one question in mind as you read: what would you change about a general-purpose model if you wanted it to do one specific job reliably? The sections below answer that question step by step, tracing what the model has learned at each stage.

Imagine you want a program that can write fluent sentences. You could try to write rules for grammar and word choice by hand, but language is too varied for that to work. A different approach is to let a model learn patterns from a large body of text and then use what it learned to generate new text. The process of building that learned starting point is called pretraining. It is the stage where a model is first trained on text before anyone asks it to do a specific job.

The general process of creating an LLM has two stages. The first is pretraining, and the second is fine-tuning. The word pre in pretraining points to the fact that it comes first. In the first stage, a model is trained on a large and diverse dataset so that it develops a broad understanding of language. The result is a pretrained model, which then serves as a foundation that can be refined further in the second stage.

The pretrained model is not thrown away after the first stage. It is the starting point for the second. Fine-tuning does not begin from nothing. It begins from a model that already has broad language knowledge, and then it adapts that model using a more focused dataset. Fine-tuning trains the pretrained model further on a narrower, labeled dataset specific to particular tasks or domains.

Read the diagram from left to right. Each stage adds something different. Pretraining adds a broad understanding of language. Fine-tuning adds focus on one job or domain. The pretrained model carries its broad knowledge forward into the second stage rather than being rebuilt from scratch.

Pretraining is described in terms of a training objective. The LLM is trained on large text datasets with the goal of predicting the next word in a text. Learning to predict the next word over a huge and varied body of text is what builds the broad understanding of language that the pretrained model carries forward.

The key contrast is in the kind of data. Pretraining works with large text datasets, and its objective is next-word prediction. Fine-tuning works with labeled data, meaning examples that come with the information the model should learn to produce or recognize for a specific purpose. In other words, pretraining asks the model to continue text, while fine-tuning shows the model what a correct output looks like for a particular task.

Before a model can be pretrained, two foundations are needed. The first is data preprocessing. Raw text is rarely in the shape a model can learn from directly, so it has to be prepared first. Preprocessing is the part of the workflow that turns collected text into training material. Because every later step depends on this preparation, it is worth understanding before you move on to the model itself.

The second foundation is the attention mechanism. The source describes it as being at the heart of every LLM. Think of attention as the part of the model that lets each position in a sequence take other positions into account when it builds its representation. Stage 1 of the course is where this mechanism is built in isolation, so that the pretraining work can use it as a working part rather than a mystery.

Fine-tuning uses labeled data, but the labels take different shapes depending on the task. The source identifies two categories of fine-tuning as the most popular. They differ in what the labeled examples look like. The diagram below shows the split, and the subsections explain each branch.

Read each column as one example. The instruction column pairs what the model is asked to do with the answer it should give. The classification column pairs a text with the class it belongs to. Both are labeled, but the label itself is a different kind of thing in each case.

There are two routes to a trained model. You can pretrain one yourself, or you can load openly available model weights that someone else has already produced. Pretraining an LLM from scratch is a significant undertaking. The source states that it demands thousands to millions of dollars in computing costs for GPT-like models. That range is wide, so the practical lesson is that full-scale pretraining is a major commitment rather than a routine exercise.

This cost is the reason the course narrows its goal. Instead of pretraining at full scale, the second stage focuses on implementing training for educational purposes with a small dataset. Loading openly available weights is an alternative route to a trained model without running the full training yourself. Both routes still require evaluating the resulting model.

The third topic in the second stage is evaluating LLMs. The source describes evaluation as essential for developing capable NLP systems. A trained model is a set of learned values, and those values alone do not tell you whether the model produces good text. Evaluation is the step that measures what the model does, so that you can compare versions and decide what to change.

Read the diagram from left to right. The first two steps belong to stage 1 and are about preparing data and understanding attention. The pretraining step belongs to stage 2. The branch for loading open weights sits beside pretraining because it is an alternative way to reach a trained model. Both routes end at evaluation, because a model needs to be measured whichever way it was obtained.