Building a Large Language Model
Emergent behavior is defined relative to training. Ask what the training explicitly targeted. If the model can perform a task that was not targeted, that task is emergent.
A Surprising Capability in GPT Models
Most people expect a system to be good at the thing it was practiced on and nothing more. A calculator trained on arithmetic does arithmetic, and a spell checker checks spelling. GPT models do not fit that expectation neatly. They can carry out translation between languages, even though translation was never the job they were trained to do.
What do you think happens?
If a GPT model was trained only to predict the next word in a piece of text, can it translate between languages?
Reveal answer
Yes. GPT models can carry out translation between languages, even though translation was never the job they were trained to do.
The training objective is next-word prediction. Translation is not named anywhere in that objective, yet the ability appears in the trained model. The rest of this article explains how that happens.
Defining Emergent Behavior
Emergent behavior is the ability of a model to perform tasks it was not explicitly trained to perform.
The definition has two parts worth reading slowly. The first is the word explicitly. The model may do something it was never directly instructed to do, and that gap is the whole point of the term. The second is that the ability is a capability the model has, not a skill a person programmed in by hand. The term describes something that shows up in the model. It is not a feature someone wrote into the code.
Emergent behavior is defined relative to training. Ask what the training explicitly targeted. If the model can perform a task that was not targeted, that task is emergent.
Tracing the Path from Prediction to Translation
Start with what the training actually asks of the model. The training task is next-word prediction. The model reads text and learns to predict which word comes next. Nothing in that objective names translation, and no step in training says that one language should become another. The training signal is about continuing text, not converting it.
The design of the model also does not start from translation. The source points out that the original transformer had both encoder and decoder blocks and was explicitly designed for language translation. GPT models use a larger yet simpler decoder-only architecture aimed at next-word prediction. So the GPT design begins from prediction, which is what makes its translation ability notable.
Read the diagram from left to right. The model is shaped by next-word prediction over multilingual text, and the translation ability appears at the end of the chain. Translation is not an input or an objective at any point along the way. That gap between what training targeted and what the model can do is the core of this article.
Tracing a Hypothetical Case
A hypothetical model is trained only on next-word prediction over text in several languages. Does its translation ability count as emergent behavior? Use the reasoning steps from the source.
- 1Identify the trained objective
The training objective is next-word prediction. Translation is not named in it, and no training step asks one language to become another.
- 2Check the data
The model sees multilingual text from diverse contexts while it learns to predict words. This is the exposure the source identifies as the source of the ability.
- 3Check whether translation was targeted
Translation was not an explicit training target. The model was not handed translation pairs and told to memorize them.
- 4Apply the definition
The model performs a task it was not explicitly trained to perform, so the translation ability is emergent behavior.
The translation ability is emergent behavior, because it is performed without having been an explicit training target.
Why Multilingual Data in Diverse Contexts Matters
The source explains that this capability is not taught explicitly during training. It emerges as a natural consequence of the model's exposure to vast quantities of multilingual data in diverse contexts. The model is not handed translation pairs and told to memorize them. Instead, it sees a large amount of text across many languages and many situations. While learning to predict words, it picks up patterns that connect languages.
Two conditions in the source matter. The data must be multilingual, and it must come from diverse contexts. A model that saw only one language would have no cross-language patterns to pick up. Data that varies in context gives the model many different settings in which the same kinds of meaning appear. That variety is the raw material for the cross-language patterns the source describes.
Trained Task Versus Emergent Task
It helps to put the two kinds of ability side by side. A trained task is one the training objective directly targeted. An emergent task is one the model can perform even though training never aimed at it. In the GPT case, next-word prediction is the trained task and translation is the emergent one.
The two columns are not a measure of how good the model is at each job. They describe where the ability came from. The trained column names what the training pushed toward. The emergent column names an ability that appeared beyond that push.
| Question | Trained task | Emergent task |
|---|---|---|
| Was it the training objective? | Yes, directly | No, never targeted |
| Where does the ability come from? | The training push itself | Exposure to multilingual data in diverse contexts |
| GPT example | Next-word prediction | Translation between languages |
| Question to ask | What did training explicitly target? | Was it untargeted but still shown by the model? |
Telling a trained task from an emergent one
Mistakes When Reading Emergent Behavior
Common mistake Treating emergent behavior as a skill someone programmed in by hand
What it looks likeAssuming a developer wrote a translation routine into a GPT model.
Why it failsThe source describes the ability as a capability the model has. It is not a feature someone wrote.
Do this insteadDescribe the ability as something that appears in the trained model from its exposure to data.
Common mistake Assuming the model was given translation pairs to memorize
What it looks likeReasoning that the model memorized matching sentences in two languages.
Why it failsThe source says the model is not handed translation pairs. It sees a large amount of multilingual text while learning to predict words.
Do this insteadExplain the ability through the patterns the model picks up from varied text during next-word prediction.
Common mistake Thinking that a single-language dataset could produce the same result
What it looks likeExpecting a model trained only on one language to translate.
Why it failsThe source names multilingual data as a necessary condition. A single language offers no cross-language patterns to pick up.
Do this insteadCheck both conditions: the data must be multilingual and drawn from diverse contexts.
Common mistake Confusing good performance with emergent behavior
What it looks likeCalling a model's skill at predicting the next word emergent behavior.
Why it failsNext-word prediction is the trained task. Emergent behavior is defined by a task the model performs without training aimed at it.
Do this insteadAsk first what the training explicitly targeted, then decide whether the ability goes beyond that.
Applying the Test to a New Task
The same reasoning applies to any task. Ask what the training explicitly targeted. Then ask whether the ability could have come from the data and the objective. If the ability was not targeted but the model still shows it, you are looking at emergent behavior.
A hypothetical model is trained only on next-word prediction over multilingual text from many contexts. Later, it is observed summarizing a passage in a language it was never asked to summarize. Is this emergent behavior? Explain your answer using the trained task, the data conditions, and the definition.
Hints
- Start by naming the trained task. What did the training objective explicitly target?
- Check whether the data was multilingual and drawn from diverse contexts.
- Ask whether the summarization was an explicit training target, then apply the definition.
Key Takeaways
- Emergent behavior is the ability of a model to perform tasks it was not explicitly trained to perform.
- GPT models are trained on next-word prediction, yet they can perform translation, which was never the training target.
- The ability emerges from exposure to vast quantities of multilingual data in diverse contexts, not from explicit instruction for each task.
- A trained task is the one the objective directly targeted; an emergent task appears beyond that targeting, so the question to ask is what training explicitly aimed at.