Understanding Foundation Models

Imagine your team has been asked to build a customer-support assistant. Someone proposes a shortcut: skip building anything from the ground up and begin with a model that already exists and already handles language well. That shortcut is the normal starting point for building with foundation models. The harder question is what you inherit when you take that starting point. Two models can both be called foundation models and still behave very differently, and the reasons are not always visible from the outside.

This article looks at the challenges that come with adapting a model you did not build, and at practical ways to reason through those challenges. The central idea is that you will often have to make good decisions with incomplete information. You do not need to know how to train a model in order to use one well, but you do need to understand the broad categories of decision that shape a model, because those categories predict how it will behave and how it can be adapted.

The word foundation describes the role the model plays for others. It is a base that many different applications can be built on, rather than a finished product built for a single purpose. Examples named in the source include ChatGPT, Google's Gemini, and Midjourney. The source also makes clear that these models are not static: they are becoming more powerful and capable of more tasks over time.

The source describes a pattern called model as a service. Training large models requires data, compute resources, and specialized talent that only a few organizations can afford. Those organizations build the models and then offer access to them. Anyone who wants to create an AI-powered application can use these models without first investing in building a model of their own.

Building a foundation model is an expensive and intricate undertaking, and the people who know how to do it well are often bound by confidentiality agreements. The specific know-how is therefore rarely published, which means an application builder usually cannot inspect the full recipe behind a model. The source is candid that the growing lack of transparency in training makes it hard to know every design decision that went into a given model. This is why adapters must reason about what they cannot see.

Scaling up AI models increased the demand for AI applications while lowering the barrier to building them, because the costly training has already been done by others. The source summarizes this as two forces at work. More people and teams want to use AI to raise productivity, create economic value, and improve quality of life. Meanwhile, the up-front cost of getting started has dropped, because the expensive part, training the model, has already been done by someone else.

Scale matters to anyone who builds software because it changes two things at once. Models become more capable and can handle a wider range of tasks, which opens up more applications. At the same time, the effort needed to train such models is concentrated in a small number of organizations. The rest of the field has to decide how to make use of what those organizations produce. The AI models behind products such as ChatGPT, Gemini, and Midjourney are large enough that they consume a meaningful share of the world's electricity, and their developers face the possibility of running short of publicly available internet data for training.

Foundation models did not appear suddenly. The source traces their lineage: foundation models emerged from large language models, which in turn originated as language models. Applications such as ChatGPT and GitHub's Copilot can look like they arrived overnight, but they are the result of decades of technical progress. The first language models emerged in the 1950s.

The progression is one of growth rather than replacement. A language model that has grown to a very large scale is a large language model, and a large-scale model offered for others to build on is a foundation model. The same underlying idea is present throughout, but each stage describes a different size of model and a different way it reaches its users.

Language models have existed for a long time, but they have only been able to grow to the scale they are today with self-supervision. Self-supervision is the technique the source identifies as enabling this growth. The source introduces it briefly, as part of a short overview of what language models and self-supervision mean, so the key idea to hold onto is this: self-supervision lets a model learn from raw data without a person having to label every example.

To see how that can work in principle, consider an illustration that is not taken from the source. Suppose a model is given a sentence of ordinary text. The developer hides one word and asks the model to predict it. The hidden word is the training target, and it came from the text itself, so no human had to annotate it. When the model's guess is compared with the hidden word, the difference is used to adjust the model. Repeating this across enormous amounts of raw text is one way raw text becomes training signal.

A useful way to make the adaptation problem concrete is to follow a model's life in order. It begins with the data it learns from. Its structure and size are fixed by the developer. Its training is often split into a first phase that builds capability and a second phase that shapes behavior. Finally, when you ask it something, it selects an output from the many possibilities it could produce. Each stage leaves a fingerprint on what you eventually see.

The question to keep in mind at each stage is the same: if this stage were different, what would the model look like to someone using it? Answering that question turns a vague sense that a model is good or bad into a reasoned judgment about what to expect and what to change. The four decisions are training data, architecture and size, post-training, and sampling.

A model is a learner, and what it learns comes from its training data. Because models learn from data, their training data reveals a great deal about their capabilities and their limitations. The collection a developer chose, and the proportions in which different kinds of material appear in it, both matter. If a model is strong in one domain and weak in another, the imbalance is often worth checking against what it was exposed to.

Model architecture refers to the structure of a model, and choosing that structure is a key decision in building a foundation model. The source notes that the transformer architecture is currently dominant. That dominance can make architecture look settled, which invites the question of why this design has lasted and whether another design will eventually replace it. The source raises these questions as open ones and treats them as worth understanding.

Model size is the other half of this decision. Whenever a new model is released, one of the first things people ask is how large it is. Size is therefore not only a technical property but a first-glance signal that shapes expectations. For an adapter, the practical lesson is to treat architecture and size together as the structural commitments a model carries into every application built on it.

Training is often described as two stages. The first stage, pre-training, makes a model capable. Capability is not the same as safety or ease of use: a pre-trained model may know a great deal and still be difficult to work with. Post-training is the stage that addresses this gap, and its stated goal is to align the model with human preferences.

The source raises a deeper question: what exactly counts as human preference, and how can it be represented in a form a model can learn from? Those questions remain part of the problem rather than solved details. What the source does establish is that the way a developer aligns the model has a significant impact on its usability. When a model feels cooperative, cautious, or awkward to direct, post-training is one of the most likely reasons.

Sampling is the process by which a model chooses one output from all the options it could produce. It is easy to overlook. Most people attribute a model's behavior to training, yet the source argues that sampling is one of the most underrated concepts in AI. It is the step that turns the model's internal possibilities into the single response you actually read.

The source makes two claims that matter for adapters. First, sampling can explain many behaviors that otherwise seem baffling, including hallucinations and inconsistencies. If the same question produces different answers on different runs, the selection step is a prime suspect. Second, choosing the right sampling strategy can significantly improve a model's performance with relatively little effort. That combination makes sampling a rare lever: it can change results substantially without retraining anything.

When a model misbehaves, the source suggests diagnosing by symptom. Match run-to-run variation to sampling, domain errors to training data, and conduct problems to post-training. The worked example below applies that rule to a support assistant.