Evaluating Large Language Models

Many people have used a phone keyboard that suggests the next word as they type. You begin a message, and the keyboard offers a few words that are likely to come next. That habit of guessing what follows a given stretch of text is the same habit at the heart of a large language model. This article uses that habit as the starting point for understanding what an LLM is, what the word large means, and how training turns simple guesses into a capable system.

The article also places this idea inside a larger plan. Building a large language model from scratch is not one task but a sequence of connected tasks, and a good way to begin is to ask what must be ready before anything else can happen. Keep that question in mind as you read.

The chapter's plan can be divided into three parts, each with a clear purpose. Stage 1 covers the fundamental data preprocessing steps and the attention mechanism. Stage 2 moves from understanding to construction: it teaches how to code a GPT-like LLM and pretrain it so that it is capable of generating new texts. The third part concerns evaluation of LLMs, which is described as essential for developing capable NLP systems.

The diagram is a simplified reading of the plan rather than a technical design. Its purpose is to show the direction of travel: preparation and the attention mechanism lead into building and pretraining, and building leads into evaluation. Read this way, the roadmap is less a list of unrelated topics and more a path in which each stage prepares the ground for the next.

Data preprocessing is the first topic the plan names, and it is marked as core material. The plan describes it as learning the fundamental steps of preprocessing data for LLMs. The word fundamental matters here: the plan is teaching the basic steps that every such effort relies on, not a specialized trick for one model.

The attention mechanism is introduced in the same first stage as preprocessing and is described as being at the heart of every LLM. Because the later stages build a model and then pretrain it, the prepared data and the attention mechanism are what those later stages depend on. Preprocessing therefore sits at the start of the roadmap because nothing downstream can happen until the material is ready.

Three parts of that definition do most of the work. The first is that an LLM is a neural network, which means it is a learned mathematical system made of many connected adjustable pieces, not a hand-written list of rules about grammar. The second is that it handles text, so its inputs and outputs are language. The third is that it is designed to do this in a humanlike way, meaning its output is meant to read naturally and respond to what was written.

The phrase deep neural network signals that the network has many layers of processing. That depth is what allows the model to pick up subtle patterns across a long passage instead of just adjacent words.

An LLM is trained on massive amounts of text. In some cases that training text can cover large portions of the entire publicly available text on the internet. A model that only saw a few thousand sentences could not learn the wide variety of patterns that the largest models pick up.

The word large in the name carries two separate meanings. Both are about scale, but they measure different things. Once you can tell them apart, it becomes easier to reason about what makes a model powerful and what goes into building one.

Parameters are the adjustable weights inside the network. Imagine a large mixing board in a recording studio with thousands of sliders. Each slider sets how much one signal contributes to the final sound. Training is the process of moving the sliders until the output matches what you want. In an LLM, the sliders are the parameters, and the thing you want the output to match is the next word in a sequence.

The mixing board is an analogy, not a technical description. What the source establishes is that parameters are optimized during training so the model can predict the next word in a sequence. Starting from that, the next question is why next-word prediction is the task used to set them.

The task used to train these models is next-word prediction. The model is shown a sequence of words and asked to guess the word that comes next. Language is naturally sequential: each word arrives after the ones before it, and meaning builds up as the sequence unfolds. Next-word prediction takes advantage of that order, so that learning to guess well forces the model to pick up context, grammatical structure, and the relationships between words.

The model does not need a teacher to label each sentence with its meaning. The next word in the text itself serves as the answer. This is why the text can be so plentiful: every position in a passage becomes a small prediction exercise.

A natural reaction is suspicion. Guessing the next word sounds far too modest to produce a system that can understand and respond to text. Yet this is one of the more surprising facts in the field: a task this simple has produced models that are remarkably capable. The source notes that this surprised many researchers.

Read the training loop from left to right. The text supplies context, the model makes a guess, and the guess is compared with the word that actually follows in the text. The result feeds back into the parameters, nudging them so that future guesses improve. Repeat this across an enormous amount of text and the parameters gradually settle into values that capture a great deal about language.

This article only defines the concept and the task. The source notes that the step-by-step training procedure is covered in later chapters, so treat the loop above as a map of the idea rather than a full implementation.