The Training Loop for Next-Word Prediction

Many people have used a phone keyboard that suggests the next word as they type. You type the start of a message, and the keyboard offers a few words that are likely to come next. That habit of guessing what comes after a given stretch of text is the same habit at the heart of a large language model. The difference is one of scale and ambition. An LLM is built to handle a far wider range of language, and it learns that skill from a very large body of text rather than from a small list of common phrases.

This article builds a definition from the ground up. We start with what an LLM is, then look at what the word large is doing in its name, then follow the training task that shapes the model. By the end you should be able to explain the definition in your own words and see why a surprisingly simple guessing task matters so much.

Three parts of that definition do most of the work. The first is that an LLM is a neural network, which means it is a learned mathematical system made of many connected adjustable pieces. It is not a hand-written list of rules about grammar. The second is that it handles text, so its inputs and outputs are language. The third is that it is designed to work in a humanlike way, meaning its output is meant to read naturally and respond to what was written.

The phrase deep neural network signals that the network has many layers of processing. The source connects this depth to the model's ability to pick up subtle patterns across a long passage, rather than only relationships between adjacent words.

The word large in the name carries two separate meanings. Both are about scale, but they measure different things. Once you can tell them apart, it becomes easier to reason about what makes a model powerful and what goes into building one.

The first meaning is the number of parameters in the model. Parameters are the adjustable weights inside the network, so this measures how many adjustable pieces the structure contains. The second meaning is the immense dataset used to train the model. This measures how much text the model learns from. The two are related because a model is built with many adjustable pieces and then shaped by a large amount of text, but they are still different quantities.

The scale of the data matters for a practical reason. A model that only saw a few thousand sentences could not learn the wide variety of patterns that the largest models pick up. This is part of what distinguishes these models from smaller language systems.

A worked example makes the training loop concrete. Pick one short stretch of text and trace one prediction from start to finish. The example below is an illustration we invented to show the process. It is not a sample taken from the source material.

Read the flow from left to right, as the source suggests. The text supplies context, the model makes a guess, and the guess is compared with the word that actually follows in the text. The result feeds back into the parameters. A single pass changes the parameters only slightly. The learning comes from repeating this across an enormous amount of text.

Parameters are the adjustable weights inside the network. Picture a large mixing board in a recording studio with thousands of sliders. Each slider sets how much one signal contributes to the final sound. Training is the process of moving the sliders until the output matches what you want. In an LLM, the sliders are the parameters, and the thing you want the output to match is the next word in a sequence.

The mixing board analogy also explains why one slider movement is not the whole story. A single adjustment changes one small aspect of the sound. The overall result emerges from all the sliders together, and the parameters gradually settle into values that capture a great deal about language.

Next-word prediction works as a training task because language is sequential. Each word arrives after the ones before it, and meaning builds up as the sequence unfolds. Next-word prediction takes advantage of that order. Learning to guess well forces the model to pick up context, grammatical structure, and the relationships between words.

A natural reaction is suspicion. Guessing the next word sounds far too modest to produce a system that can understand and respond to text. Yet this is one of the more surprising facts in the field. A task this simple has produced models that are remarkably capable, and the source notes that this surprised many researchers.

Training is a cycle. The model is given context, makes a prediction, measures how far that prediction is from the actual next word, and adjusts the parameters. Then the cycle begins again with the next example. Every position in a passage gives one small exercise, so the same loop runs over a very large number of examples.

Repeating this across an enormous amount of text is what gradually shapes the parameters. No single prediction teaches the model much on its own. The accumulated adjustments from many predictions are what capture patterns of language.

The ideas in this article are easy to blur together. The most common errors come from mixing up what the model learns from, what the model is made of, and what the training task is.