Foundation Models

Imagine trying to teach a system to write fluent English without giving it a single answer key. There is no list of correct responses to copy and no teacher marking each sentence. Yet large language models are built exactly this way at their starting point. Before a model is asked to classify emails, summarize articles, or act as a personal assistant, it first goes through a broad, general learning phase on a huge amount of plain text. Only after that does it get shaped toward a particular job.

The question this article answers is what happens during that first phase, and how it sets up everything that comes after. As you read, keep track of two things at each step: what data the model is looking at, and what it is being asked to produce.

The first step is to train the model on a large corpus of text data, often described as raw text. The word raw has a specific meaning here: the text is just regular writing, with no labeling information attached. Nobody has tagged each document with a category, a sentiment, or a correct answer. The text itself is the only material the model receives.

Raw text is not always used in its original form. Filtering may be applied before training, for example by removing formatting characters or by discarding documents written in languages the training process does not handle. Filtering changes what the corpus contains, but it does not add labels. The data remains unlabeled.

Readers who have studied traditional machine learning will expect labels to be necessary. In conventional supervised learning, each training example comes with a target answer, and the model learns by comparing its guesses against those answers. Labeling information is typically required for traditional machine learning models and for deep neural networks trained with the conventional supervised paradigm.

Pretraining does not work that way. LLMs use self-supervised learning, a paradigm in which the model generates its own labels from the input data. The data is not hand-labeled by people, yet the training process still has a target to aim for. The labels are produced from the text itself.

The diagram below traces how one short passage yields many training pairs. Each word becomes an input, and the word that follows it becomes the target. Notice how the target at each step is already present in the text, so no person has to write it down.

Pretraining is a repeated cycle. The model reads a piece of text, makes a guess about the next word, compares that guess with the word the text actually contains, and adjusts its internal weights so that the next guess is a little better. The loop below shows one cycle and how it repeats across the corpus.

After pretraining, the model is not yet specialized for a particular purpose. The second stage, fine-tuning, takes the pretrained model and trains it further on a narrower dataset that is more specific to particular tasks or domains. Fine-tuning is therefore a continuation of training, not a fresh start.

The data changes at this stage. Pretraining uses a large corpus of raw, unlabeled text. Fine-tuning uses a smaller labeled dataset, in which the examples are tied to the specific task the developer wants the model to perform. The contrast is shown below, with the two stages side by side.

A pretrained LLM is called a base or foundation model. GPT-3 is a typical example. The source describes it as capable of text completion, meaning it can finish a half-written sentence that a user provides. It also has limited few-shot capabilities, which means it can learn to perform a new task from only a few examples rather than requiring extensive training data. The source notes that GPT-3 is the precursor of the original model offered in ChatGPT.

This explains why the pretrained model is called a foundation. It is useful on its own for some behaviors, such as completing text, but it is also the thing that gets refined. Its broad abilities are the raw material that fine-tuning shapes toward tasks like classification, summarization, or translation. One base can branch into many specialized models, as the hierarchy below shows.