Coding a Text-Generating LLM

Most people have used a chat-based AI assistant that can answer a question, rewrite a paragraph, and render a passage into another language, all within one conversation. It is natural to assume that such a system must be enormously complicated, and that the only way to understand it is to study a finished product from the outside. This article takes a different route. Instead of treating the assistant as a sealed box, it asks what the smallest useful piece is and how the larger behavior is put together from it.

The course section this article belongs to proposes that you can understand a complex assistant by constructing one yourself, one layer at a time. The layers are a text generator at the bottom and a set of query techniques on top of it. Keep that two-layer picture in mind, because the rest of the article fills it in.

The foundation of everything in this part of the course is an LLM that can generate text. The source material emphasizes two points. First, an LLM can generate texts. Second, an LLM can be coded to generate texts. The second point matters because it turns an abstract capability into something you can build and run yourself. Generation is therefore not just a property you observe in a finished product; it is the first thing you construct.

This article does not explain the internal mechanics of how the generator produces each piece of its output. Its goal is to show where generation sits in the overall design, and why every later task is built on top of it.

A query technique is a way of getting an LLM to carry out a particular job. The generator provides the general ability to produce text, and a query technique points that generator at a specific task. Query techniques direct the generator toward specific jobs, and this part of the course covers three of them.

The diagram can be read in two directions, and each direction is useful. Reading it from the bottom up gives the order of study: first you learn how a generator produces text, then how to point that generator at specific tasks, and finally how those tasks combine into an assistant. Reading it from the top down gives a debugging habit. When an assistant behaves unexpectedly, ask whether the problem lies in the generator underneath or in the technique that is directing it.

The source names three jobs that this part of the course focuses on: answering questions, summarizing text, and translating text. Each one asks the generator for a different kind of output from the same kind of input. The comparison below gives a short description of each, along with the kind of request it fits.

Matching a request to its technique is a practical habit that follows from the layered view. The habit has two steps: look at what goes in, then look at what should come out. Before you read the answer, try the prediction below.

The layered view also tells you where to look when something goes wrong. Locating a problem in either the generator or the technique is itself a practical habit. A useful question to ask is whether the assistant is failing at producing text at all, or whether it is producing text that is aimed at the wrong job.