Autoregressive Language Models

Large language models can seem to do many different things. They answer questions, summarize documents, and continue a story. It is natural to assume that each of these abilities must be built in separately, with its own training and its own machinery. The GPT family takes a different starting point. Its pretraining rests on one deliberately simple goal, and the other abilities grow out of it.

Keep this single sentence in mind, because the rest of this article unpacks it. We will look at what one prediction looks like, where the architecture comes from, and why the model's own outputs become part of its later inputs.

Begin with a partial sentence, such as The cat sat on the. The model sees only these words. Its job is to produce a guess for the word that comes next. A plausible continuation is mat, although a model might weigh several candidates before settling on one.

What matters for the architecture is the input-output shape of this step. The words so far go in, and one next word comes out. The rest of the design is about making that mapping from context to next word as good as possible.

The source also notes that this approach helps the model understand how words and phrases typically fit together in language. That understanding is described as a foundation that can be applied to various other tasks. The next-word objective is the pretraining step, and its benefits are meant to carry over beyond that single prediction.

The source compares GPT with the original transformer architecture covered earlier in the course. Its verdict is that the general GPT architecture is relatively simple. The simplification is specific. GPT is essentially the decoder part of the original transformer, with the encoder removed.

Reading the comparison from top to bottom shows the relationship. The original transformer has two named parts. GPT takes only the decoder side, so the encoder does not appear in its design. This is why it is fair to describe GPT as a decoder-only design, with the decoder serving as the entire architecture.

Decoder-style models like GPT generate text by predicting one word at a time. Because of this, they are considered a type of autoregressive model. An autoregressive model is defined as one that incorporates its previous outputs as inputs for future predictions.

The important word in that definition is previous. A model that produces a word does not throw that word away. The word becomes part of what the model reads when it works out the next one. This is the feature that gives the label its meaning.

Read the cycle as a loop. The words so far feed the prediction step. The chosen word is then added to the sequence, and the longer sequence is what the next prediction reads. This loop is the mechanism behind the autoregressive label, and it is the reason GPT's word choices depend on everything that came before.

The source notes that because each new word is chosen based on the sequence that precedes it, the resulting text gains coherence. Each word is anchored to the words already written rather than being picked in isolation.