Decoder-Only Models and Next-Word Prediction
The objective is local and simple: given the text so far, predict the next word. Everything the model produces, including long outputs, is built from repeated applications of that one objective.
A Model Trained for One Job
Most people expect a system to be good at the thing it was practiced on and nothing more. A calculator trained on arithmetic does arithmetic. A spell checker checks spelling. It is therefore surprising to learn that GPT models can carry out translation between languages, even though translation was never the job they were trained to do.
What do you think happens?
A GPT model is trained only to predict the next word in a piece of text. Can it still carry out translation between languages?
Reveal answer
Yes. GPT models can carry out translation between languages, even though translation was never the training target.
The rest of this article explains how a training objective about continuing text can lead to an ability the objective never named.
The answer to the prediction is yes, and the reason is worth working through step by step. The sections that follow first examine what the training actually asks of the model, then define the term emergent behavior, and finally trace how a skill the model was never directly taught can appear in its output.
What Next-Word Prediction Asks of the Model
The training task is next-word prediction. The model reads text and learns to predict what word comes next. Nothing in that objective names translation, and no step in training says that one language should become another. The training signal is about continuing text, not converting it.
A GPT model produces output by repeating one step. It predicts a next word, adds that word to the text, and then predicts again from the longer text. The diagram below shows this loop. Each predicted word becomes part of the context for the next prediction, so a single-word predictor can build up a whole sentence one word at a time.
The objective is local and simple: given the text so far, predict the next word. Everything the model produces, including long outputs, is built from repeated applications of that one objective.
Defining Emergent Behavior
Emergent behavior is the ability of a model to perform tasks it was not explicitly trained to perform.
The definition has two parts that matter. The first is the word explicitly. The model may do something it was never directly instructed on, and that is the point of the term. The second is that the ability is a capability the model has, not a skill a person programmed in by hand. The term describes what shows up in the model, not a feature someone wrote.
Emergent behavior is identified by a gap. A task is emergent when the model can perform it but training never aimed at it. If you find a capability and cannot point to any part of training that targeted it, you are looking at emergence.
Where the Translation Ability Comes From
The source explains that this capability is not explicitly taught during training. It emerges as a natural consequence of the model's exposure to vast quantities of multilingual data in diverse contexts. The model is not handed translation pairs and told to memorize them. Instead, it sees a large amount of text across many languages and many situations, and while learning to predict words, it picks up patterns that connect languages.
Two conditions in this explanation carry the argument. The data is multilingual, and it comes from diverse contexts. A model that saw only one language would have no cross-language patterns to pick up, so multilinguality is needed. Data that varies in context gives the model many different settings in which the same kinds of meaning appear. That variety is the raw material for the patterns that make translation possible.
The design of the model also matters. The source notes that the original transformer had both encoder and decoder blocks and was explicitly designed for language translation. GPT models use a larger yet simpler decoder-only architecture aimed at next-word prediction. The design of a GPT model therefore does not start from translation at all, which is what makes its translation ability notable.
Tracing Translation Through Repeated Prediction
The following trace follows one hypothetical scenario from prompt to output. The scenario is written to show the logic, while the claims about the model come from the source. Read the flowchart from top to bottom, then work through the steps.
Tracing a Translation Request
A model trained only on next-word prediction over multilingual text receives the prompt 'Translate to French: cat' and must produce the French word.
- 1Identify what training targeted
The training objective asked only for the next word in text. It did not name translation, and no step said one language should become another.
- 2Read the prompt as text
The prompt is a sequence of words. The model processes it the same way it processes any text it has seen during training.
- 3Predict the next word
The model predicts a word that fits the text so far. The word that comes after the instruction and the English word is a French word, so the prediction continues the text in French.
- 4Append and repeat
The predicted word is added to the text, and the model predicts again from the longer text, as the loop in the previous section describes.
- 5Compare with the trained objective
The sequence of predictions produced a translation even though no step in training was a translation step. The ability is emergent.
The output 'chat' appears through repeated next-word prediction. The translation ability was not the training target, so it is an emergent task.
Trained Task Versus Emergent Task
It helps to put the two kinds of ability side by side. A trained task is one the training objective directly targeted. An emergent task is one the model can perform even though training never aimed at it. In the GPT case, next-word prediction is the trained task and translation is the emergent one.
| Aspect | Trained task | Emergent task |
|---|---|---|
| Definition | A task the training objective directly targeted | A task the model can perform though training never aimed at it |
| GPT example | Predicting the next word in text | Carrying out translation between languages |
| Where the ability came from | The training objective pushed toward it | It showed up beyond that push, from exposure to data |
| Question to ask | Did training explicitly target this? | Was it untargeted, yet the model still shows it? |
Trained and emergent tasks differ in where the ability came from, not in how good the model is at each job.
The gap between the two columns is the central point. The left column names what training pushed toward. The right column names an ability that appeared beyond that push. The source attributes that gap to exposure to multilingual data in diverse contexts, not to a second objective added to training.
Common Mistakes When Reading Emergent Behavior
Common mistake Assuming the model was given a separate translation feature
What it looks likeBelieving that engineers wrote a translation module inside a GPT model.
Why it failsThe source says the ability is not explicitly taught. It emerges from exposure to data during training.
Do this insteadDescribe translation as an emergent capability that arises from next-word prediction over multilingual text.
Common mistake Treating emergent behavior as a sign the model is good at everything
What it looks likeConcluding that a model which translates must also be strong at every other task.
Why it failsEmergence is defined by a gap between training and ability. It says nothing about how well the model performs each job.
Do this insteadAsk separately whether the task was trained and whether the model performs it well.
Common mistake Ignoring the data conditions
What it looks likeSaying that any training data is enough for translation to emerge.
Why it failsThe source names two conditions, multilingual data and diverse contexts. A model that saw only one language would have no cross-language patterns to pick up.
Do this insteadName both conditions when explaining why emergence is possible.
Common mistake Confusing the training objective with the final output, since both are about words
Applying the Reasoning to a New Task
Suppose a large language model is trained only on next-word prediction over text in many languages from many kinds of sources. Someone later notices it can summarize a paragraph, even though summarization was never its training target. Using the reasoning in this article, explain whether summarization is a trained task or an emergent task, and say which part of the training story supports your answer.
Hints
- Start by asking what the training objective explicitly targeted.
- Then ask whether the ability could have come from the data and the objective.
- Check whether the ability was targeted or only shown by the model.
Key Takeaways
- Emergent behavior is the ability of a model to perform tasks it was not explicitly trained to perform.
- A GPT model is trained on next-word prediction, yet it can carry out translation, which was never the training target.
- The translation ability emerges from exposure to large amounts of multilingual data in diverse contexts, not from instruction for each task.
- A trained task is one the objective directly targeted; an emergent task appears beyond that targeting, so the question to ask is where the ability came from.