Introduction to Large Language Models and the Book

This section of the book is short. On page 23 it introduces a plan for building a large language model, an LLM, from scratch. The phrase from scratch is the key to reading it. The book does not stop at describing a finished model, and it does not only show how to call a model through an existing service. It sets out to take the learner through the construction itself, starting from basic pieces and working toward a model the learner has built.

Reading a plan well means asking what it commits the reader to. Here the commitment is to building rather than only observing. A learner who reads the plan as a promise of construction will know what to look for in the chapters that follow: concrete parts of a model assembled in order, with each part understood as it is added.

A useful way to read this section is to follow the state of the learner's understanding as the chapter proceeds. The learner moves through three states. Each state is distinct, and keeping them apart prevents a common confusion between a plan and the thing the plan produces.

The word architecture here refers to how the parts of the network are organized. It does not refer to a single formula. For a beginner, the two facts that matter are these: transformer architectures are the design used in large language models, and attention mechanisms are a key component inside them. Attention is not a separate system bolted onto a language model. It is one of the central parts of the transformer design.

Language arrives as a sequence. A sentence is a run of words, and the meaning of a later word often depends on something said much earlier. The source material flags the modeling of long sequences as a real problem that has to be handled deliberately rather than ignored. Attention is the family of mechanisms that addresses this.

The key word is focus. A model that reads a sequence does not need to treat every earlier position as equally important for every decision it makes. Attention lets the model give more weight to some parts of the input than to others. The source material states the purpose directly: attention mechanisms handle long-range dependencies in input sequences, meaning relationships between parts of the input that are far apart from each other.

Read the flow from left to right. The input sequence enters the attention mechanism, which scores the parts of the sequence against one another. Those scores produce a weighted focus, where some parts count for more than others, and that focus shapes what the model produces. This describes the flow of information. The source material treats the detailed computation of these weights as a later topic.

Self-attention is the form of attention in which the sequence is compared against itself. Each position in the input can look at other positions in that same input and decide how much they matter. This is the mechanism connected to the long-sequence problem: because any position can draw on any other position, dependencies spanning a long stretch of text need not be lost along the way.

Self-attention alone lets a position look at any other position. Causal attention adds a restriction. The source material says causal attention is used to prevent the model from attending to future tokens. A token may draw on what came before it, but not on what has not yet been produced. This restriction matters most when the model generates text, because at generation time the future has not been written yet.

The source material states that autoregressive models are used in large language models. Their defining feature is a feedback loop. The model produces an output, that output becomes part of what the model conditions on, and the model then produces the next output. Each new output is therefore shaped by the outputs that came before it.

Follow the loop around. The model reads the previous outputs, produces a next output, and that output is appended to the record of previous outputs. The cycle then repeats. The return arrow is the essential part. Without it, the model would produce one output from the input and stop. Autoregressive generation is what turns a single prediction into an extended sequence of text.

Large language models are trained using large datasets and various training methods. The source material states as an important fact that large language models require large datasets for training. Before training can happen, the data itself has to be made usable, and the source material identifies this step as crucial.

The scale is substantial. The source's reference list includes a resource titled Dolma: An Open Corpus of Three Trillion Tokens for LLM Pretraining Research, which gives a sense of how large the text collections used for pretraining can be. The word pretraining points to the broader pattern: a large model is first trained on a large general dataset, and it can then be adapted further.

A pre-trained model has already learned from a large dataset. Fine-tuning starts from that model and adapts it to a narrower purpose. The source material's index contains a clear case: fine-tuning for classification, which includes using an LLM as a spam classifier. The general model is the starting point, and the fine-tuned model is the specialized result.