Building a Language Model from Scratch

Many people have used a phone keyboard that suggests the next word as they type. You type the start of a message, and the keyboard offers a few words that are likely to come next. That habit of guessing what comes after a given stretch of text is the same habit at the heart of a large language model. The difference is one of scale and ambition: an LLM is built to handle a far wider range of language, and it learns that skill from a very large body of text rather than from a small list of common phrases.

Three parts of that definition do most of the work. The first is that an LLM is a neural network, which means it is a learned mathematical system made of many connected adjustable pieces, not a hand-written list of rules about grammar. The second is that it handles text, so its inputs and outputs are language. The third is that it is designed to do this in a humanlike way, meaning its output is meant to read naturally and respond to what was written.

The phrase deep neural network signals that the network has many layers of processing. That depth is what allows the model to pick up subtle patterns across a long passage instead of only the words that sit right next to each other.

The word large in the name carries two separate meanings. Both are about scale, but they measure different things. Once you can tell them apart, it becomes easier to reason about what makes a model powerful and what goes into building one.

The first meaning is the number of parameters in the model. The second is the immense dataset used to train it. The parameters are the size of the structure; the dataset is the size of the experience that shapes it. A model with many parameters but very little text to learn from has a structure with little to hold, and a large body of text fed into a tiny network has little capacity to store what it reads.

Parameters are the adjustable weights inside the network. Picture a large mixing board in a recording studio with thousands of sliders. Each slider sets how much one signal contributes to the final sound. Training is the process of moving the sliders until the output matches what you want. In an LLM, the sliders are the parameters, and the thing you want the output to match is the next word in a sequence.

The task used to train these models is next-word prediction. The model is shown a sequence of words and asked to guess the word that comes next. Language is naturally sequential: each word arrives after the ones before it, and meaning builds up as the sequence unfolds. Next-word prediction takes advantage of that order, so learning to guess well forces the model to pick up context, grammatical structure, and the relationships between words.

Notice what this means in practice. The model does not need a teacher to label each sentence with its meaning. The next word in the text itself serves as the answer. This is why the text can be so plentiful: every position in a passage becomes a small prediction exercise.

A worked example makes the training loop concrete. Follow one prediction from start to finish. The sentence fragment below is an illustration of the process, not a sample of what the source discusses.

A single correction is not the whole story. The source describes the process as a loop: read the diagram from left to right, and then repeat the cycle across an enormous amount of text. Each position in a passage yields its own prediction and its own signal, and each pass moves the parameters a little. Over many passes the parameters gradually settle into values that capture a great deal about language.

An LLM is a deep neural network that works with humanlike text. Its size is described two ways: by the number of adjustable parameters and by the immense dataset it learns from. Its parameters are tuned by a simple task, predicting the next word, in which the text supplies its own answers. Each guess is compared with the real next word, and the resulting signal nudges the parameters. Repeated across a vast body of text, this loop shapes the model.