Transformer Encoder and Decoder Blocks

Imagine building a model whose entire training objective is to guess the next word in a passage of text. Nobody gives it translation exercises, bilingual dictionaries, or labeled pairs of sentences in two languages. Its only job is continuing text, one word at a time. The question this article works through is whether such a model could later turn a sentence from one language into another.

The defining feature is the word explicitly. An emergent ability is not taught as a direct target during training. Instead it appears as a byproduct of other learning. In the case of GPT models, the ability to translate is not something the training process aims at. It arises because the model is exposed to a very large amount of text that includes many languages, appearing in many different contexts.

The original transformer was built with two halves: an encoder and a decoder. The encoder reads an input sentence, and the decoder produces the output sentence. This design was made explicitly for language translation, so translation was the purpose the architecture was shaped around from the start.

GPT models take a different route. They use a decoder-only architecture, which is simpler in structure than the original two-part design, and their training aim is next-word prediction. The architecture is larger in scale, but it is not built around translation. The surprise is that a model shaped for one narrow purpose ends up able to handle a task outside that purpose.

Follow the path that information takes. The training data contains large amounts of text in many languages, used in many different situations. The only training signal is next-word prediction. The source describes the translation ability as a natural consequence of exposure to this multilingual data. When a model has seen many languages used in many situations, patterns that connect one language to another become part of what it has absorbed. Translation then becomes something the model can carry out when asked, even though no translation lesson was part of its training.

Notice what the diagram does not include. There is no translation task node feeding into training. The translation ability sits at the end of the path as an outcome. The source does not describe further internal mechanics of how the patterns are stored, so this article does not claim any.

It helps to set emergent behavior beside the more familiar case, in which a model is trained directly on a task. In direct training, the task appears in the training objective and the model is scored on it. In emergent behavior, the task is absent from the objective, and the capability shows up anyway.

The practical consequence is the main benefit the source highlights. If a single large generative language model can pick up tasks it was not trained for, then one model can serve many purposes. A separate specialized model does not have to be built, trained, and maintained for each task. The source states this directly: diverse tasks can be performed without using diverse models for each.

Several capabilities can therefore share one training objective and one set of learned patterns. Translation is the only task the source names explicitly. The general claim is that the same model is not limited to a single job.