Next-Word Prediction in Language Models

The surprise is that a model shaped for one narrow purpose, predicting the next word, ends up able to handle a task outside that purpose. The gap between what the model was built for and what it can later do is the central puzzle this article addresses.

Programming

A Model Trained Only to Continue Text

Imagine building a language model whose entire training objective is to guess the next word in a passage of text. Nobody gives it translation exercises, bilingual dictionaries, or labeled pairs of sentences in two languages. Its only job is continuing text, one word at a time.

What do you think happens?

Could a model trained only to guess the next word later turn a sentence from one language into another?

No, because it was never shown translation examples
Yes, because translation ability can appear without being a training target

Reveal answer

Answer:

Yes. GPT models, which are trained on next-word prediction, can translate even though translation is not what their training aims at.

The source explains that the translation ability arises as a natural consequence of exposure to vast quantities of multilingual text. The point to hold onto is not the answer alone but the reasoning: a capability can show up even when it was never the target of training. The rest of this article explains why this is less strange than it first looks.

The Original Transformer's Two Halves

The original transformer was built with two halves: an encoder and a decoder. That design was made explicitly for language translation. The encoder reads an input sentence, and the decoder produces the output sentence. Translation was therefore the purpose the architecture was shaped around from the start.

This matters for the rest of the story. When a model's structure is designed around one task, you would expect its abilities to be tied to that task. The original transformer's translation ability follows directly from the way its two halves were built and what they were built to do.

How GPT Models Differ in Design

GPT models take a different route. They use a decoder-only architecture, which is simpler in structure than the original two-part design, and their training aim is next-word prediction. The architecture is larger in scale, but it is not built around translation.

Key point

Defining Emergent Behavior

Definition

An emergent behavior is an ability a model shows without having been explicitly trained to perform that task.

The defining feature is the word explicitly. An emergent ability is not taught as a direct target during training. Instead, it appears as a byproduct of other learning. In the case of GPT models, the ability to translate is not something the training process aims at. It arises because the model is exposed to a very large amount of text that includes many languages, appearing in many different contexts.

Tracing the Path From Prediction to Translation

Follow the chain of events that produces the ability. Each step depends on the one before it, and none of them is a translation lesson. The diagram below traces the path from the training text to a model that can translate when asked.

What happens between the training objective (predict the next word) and the later ability to translate a sentence the model was never explicitly taught to translate?

Read the diagram from top to bottom. The training text is the input. The objective of predicting the next word is the only thing the training process aims at. The resulting decoder-only model is shaped by that objective. Through exposure to multilingual text, patterns that connect one language to another are absorbed. When someone asks the model to translate, that absorbed material is what the model can draw on. The translation step is not a stage that anyone trained directly.

Why Multilingual Exposure Matters

The source describes the translation ability as a natural consequence of exposure to vast quantities of multilingual data across diverse contexts. The phrase across diverse contexts is important. It is not enough for a language to appear in the training text; it appears in many situations, with many kinds of surrounding material.

When a model has seen many languages used in many situations, patterns that connect one language to another become part of what it has absorbed. Translation then becomes something the model can do when asked, even though no translation lesson was part of its training. Multilingual exposure is therefore the ingredient that makes the emergent ability possible. Without text in more than one language, there would be no cross-language patterns to absorb.

In the real world

Consider a generated scenario, not drawn from the source. Suppose a training text contains a short English sentence about the weather and, elsewhere in the same corpus, a French sentence describing the same weather in different words. Across millions of such passages, the model is never told these two sentences mean the same thing. Its job is still only to predict the next word. Yet the repeated appearance of the same ideas in different languages is the kind of exposure the source describes as the seedbed of translation.

Emergent Versus Directly Trained Behavior

It helps to set emergent behavior beside the more familiar case, in which a model is trained directly on a task. In direct training, the task appears in the training objective and the model is scored on it. In emergent behavior, the task is absent from the objective, and the capability shows up anyway. The diagram places the two routes side by side.

How do the two routes differ in what data and training step produce the capability, and what does each look like side by side?

Feature	Direct task-specific training	Emergent behavior
Position of the task	Appears in the training objective	Absent from the training objective
Training signal	The model is trained and scored on the task	The model is trained on another objective, such as next-word prediction
How the ability arises	Produced by aiming training at the task	Appears as a byproduct of other learning
Example from the source	Not named in the source	GPT models translating after next-word training

Comparing the two routes to a model capability, based on the source definition.

One Model Covering Many Tasks

The practical consequence is the main benefit the source highlights. If a single large generative language model can pick up tasks it was not trained for, then one model can serve many purposes. A separate specialized model does not have to be built, trained, and maintained for each task. The source states this directly: diverse tasks can be performed without using diverse models for each.

How does one shared model serve translation and other tasks, compared with a setup where each task has its own model?

The diagram shows the shape of the idea. Training begins from text and a single objective and produces one model. That one model is then able to perform several tasks. Translation is the task named in the source. The other task box stands for the broader claim that the same model is not confined to a single job.

Common Mistakes When Reading About Emergence

Common mistake
Thinking emergent means the training data had no relevant content
Why it fails
The source ties the translation ability to exposure to vast quantities of multilingual data. The data contained the material. What was absent was a translation objective.
Do this instead
Treat emergent as a statement about the training objective, not about whether the relevant patterns were present in the text.
Common mistake
Believing the translation ability was a direct training target
Why it fails
The definition says an emergent ability is one the model was not explicitly trained to perform. Translation is described as something the training process does not aim at.
Do this instead
Describe the objective as next-word prediction and the translation ability as a byproduct of exposure.
Common mistake
Confusing the original transformer's design with a GPT model's design
Why it fails
The original transformer had an encoder and decoder built for translation. GPT models are decoder-only and trained on next-word prediction.
Do this instead
Keep the two architectures separate when explaining why a GPT model's translation ability is surprising.
Common mistake
Assuming every task a model can perform was emergent
Why it fails
The source defines emergent behavior as an ability shown without explicit training for that task. Tasks that are part of the objective fall under direct training, not emergence.
Do this instead
Check whether the task appears in the training objective before labeling the ability as emergent.

Practice: Reasoning About an Emergent Ability

MEDIUM

A team trains a decoder-only model only on next-word prediction over a large body of text in several languages. Later, they find it can translate a sentence from one language into another, even though translation was never in the training objective. Explain, in your own words, why this counts as emergent behavior and what role the multilingual text played.

Hints

Start with the definition: is translation the explicit training target here?
Think about what the training text contained and how often languages appeared in varied contexts.
Then consider whether the team needed a separate model for translation, and why or why not.

Your attempt (not saved)

HARD

Name two tasks, other than translation, that you would expect a single model to handle if its abilities are emergent in the way this article describes. For each, state whether the task was an explicit training target in your scenario and what kind of exposure might support it. Keep your answer grounded in the source's claims rather than new assumptions.