Building LLMs from Scratch

Large language models, usually called LLMs, are the systems behind many recent AI tools. The introduction to this book places their rise in public view at a specific moment: 2022 and the release of ChatGPT. From that point, the source says, LLMs have taken the world by storm and have revolutionized how many people work.

The source describes LLMs as incredibly versatile. It mentions tasks such as checking grammar, composing emails, and summarizing lengthy documents. The author attributes this versatility to the models' ability to parse and generate human-like text. According to the source, that ability matters in fields ranging from customer service and content creation to technical domains such as coding and data analysis.

The author's own route into the subject began much earlier. Years before writing the book, the author's interest in language models started in a statistical pattern classification class. That class led to a first independent project: a model and web application that detected the mood of a song from its lyrics. The introduction presents this history as the reason the author wrote a book about building these models, rather than as a technical claim in itself.

What the source does provide is the introductory text of the book. In that text, the author explains why the book was written and what historical moment it responds to. That front matter is the part this article can teach accurately, so the rest of the article follows it closely.

A common belief about LLMs is that they are a black box. They produce impressive output, but nobody can explain how that output is produced. The belief is easy to form for three reasons the source implies: these systems are very large, the public sees only their answers, and the internal workings are rarely shown to ordinary users.

The author's answer to the black box belief is that scale and understandability are separate properties. Size tells you how many adjustable values a model has. It does not, by itself, tell you whether the model can be understood. A model can be large and still be understood at a fundamental level. The concept map below shows how the author's argument separates these ideas.

The author's central claim is that the best way to understand LLMs is to code one from scratch. The author has a reason for this claim that comes from experience. Several years before writing the book, the author had to learn how to implement LLMs the hard way, working through many research papers and incomplete code repositories. The book is meant to offer a gentler path: a step-by-step implementation tutorial that covers the major components and development phases of an LLM.

The source also argues that building confidence in each part of the fundamental concepts and the underlying code is crucial. The reason it gives is practical: confidence in each part helps with fixing bugs, improving performance, and experimenting with new ideas. A learner who has built each piece can tell where a fault might sit, which is hard to do when the model is treated as a finished object.

The source describes the course of work as having a clear shape. The learner implements an LLM from the ground up. Along the way the learner codes two specific pieces by hand: the pretraining functions and the fine-tuning functions. Writing these from scratch, rather than calling a ready-made library routine, is what the source calls the hands-on approach. The pipeline below shows the order in which the source presents the stages.

Read the diagram as a description of intent, not as a complete technical map. Each box is a stage the source names. The arrows show the order of presentation. The final box is the outcome the author hopes the learner reaches, and it is the only outcome the source states directly.

The source contains several statements with different weights. Some are the author's own judgment. Others are arguments about learning that the approach is said to support. The table separates them so that a learner does not read a personal opinion as an established finding.

Misreading the hands-on approach is easy, because the phrase building from scratch can suggest several different goals. The comparison below sets each misreading beside the understanding the source intends.