Sampling Strategies for Language Models

Imagine your team has been asked to build a customer-support assistant. Someone proposes a shortcut: skip building anything from the ground up and start with a model that already exists and already handles language well. That shortcut is the normal starting point for building with foundation models. The harder question is what you inherit when you take it. Two models can both be called foundation models and still behave very differently, and the reasons are not always visible from the outside.

Building a foundation model is an expensive and intricate undertaking. The people who know how to do it well are often bound by confidentiality agreements, so the specific know-how is rarely published. An application builder usually cannot inspect the full recipe behind a model, and the growing lack of transparency in training makes it hard to know every design decision that went into a given model. This article is about reasoning well under that limitation: which decisions shape a model, how each one tends to show up in behavior, and how to diagnose a misbehaving model without seeing its full recipe.

A useful way to make the adaptation problem concrete is to follow a model's life in order. The data it learns from comes first. Its structure and size are fixed by its developer. Its training is often split into a first phase that builds capability and a second phase that shapes behavior. Finally, when you ask it something, it selects an output from the many possibilities it could produce. The last step is the one this article focuses on most, because it is the step that happens every time the model answers.

At the moment of answering, the model has not yet produced a single word. It has produced a set of candidate options, each with a score indicating how well it fits. Those raw scores are converted into a probability distribution, which says how likely each candidate is. Sampling is the step that then picks one candidate from that distribution. The diagram below traces that path for a single choice.

The key consequence is that a candidate with the highest probability is not automatically the output. Sampling is a selection from a distribution, not a lookup of the top entry. This is the first reason sampling can change what you see even when the model, its training, and its prompt stay exactly the same.

Most people attribute a model's behavior to training. When a model is accurate in one area and shaky in another, the instinct is to ask what it was trained on. The source argues that sampling is one of the most underrated concepts in AI. It is the step that turns the model's internal possibilities into the single response you actually read, and it is easy to forget because it happens after everything else has been decided.

The source makes two claims that matter for anyone adapting a model. First, sampling can explain many behaviors that otherwise seem baffling, including hallucinations and inconsistencies. If the same question produces different answers on different runs, the selection step is a prime suspect. Second, choosing the right sampling strategy can significantly improve a model's performance with relatively little effort. Together these make sampling a rare lever: it can change results substantially without retraining anything.

Sampling strategies are the settings that govern how that selection is made. Common controls work on the distribution itself. A temperature-style setting makes the distribution more concentrated or more spread out, so the model either leans harder toward its top candidates or gives less likely candidates a larger share. A top-k control keeps only the k most likely candidates and removes the rest from consideration. A top-p control keeps the smallest group of most likely candidates whose probabilities add up to at least p, again removing the remainder. These are standard terms, and the source pack does not define their exact mechanics, so treat the description here as the general idea.

The illustration below uses made-up probabilities for four candidate words after a prompt. It shows what a truncating control does to the candidate set: it removes options before the selection step runs, so the choice is made among fewer and more plausible words.

Read the diagram as a before and after pair. The two lower-ranked candidates, fish and rock, are removed from the set. The surviving candidates have their probabilities adjusted so they still form a valid distribution. The selection step then operates only on cat and dog. Changing a control like this does not alter what the model learned; it alters which of the model's existing possibilities are eligible to be chosen.

Differences between foundation models trace back to a small set of decisions: training data, model architecture and size, post-training that aligns the model with human preferences, and sampling. Each decision leaves a different fingerprint on the behavior you eventually observe. The table below summarizes what each one controls and what a builder would notice if it were different.

When a model misbehaves, the source recommends diagnosing by symptom. Match run-to-run variation to sampling, domain errors to training data, and conduct problems to post-training. The tree below turns that guidance into a first-question routine: start with the symptom you observe and follow the branch to the decision most likely responsible.

The reason application builders adapt an existing model rather than training one is practical. Training is complex, costly, and often confidential, so the specific recipe is rarely available. Adapting a model lets a team start from a capable base and concentrate effort on the application. The comparison below separates the two paths on the dimensions that matter most to a builder.

The central challenge of adaptation is making good decisions with incomplete information. The solution is not to reverse-engineer every detail of a model. It is to understand the broad categories of decision that shape a model, because those categories determine how the model is likely to behave and how it can be adapted. You do not need to know how to train a model in order to use one well.