Query Techniques for Language Models

Most people have used a chat-based AI assistant that can answer a question, rewrite a paragraph, and render a passage into another language, all within one conversation. It is natural to assume that such a system must be enormously complicated, and that the only way to understand it is to study a finished product from the outside, as if it were a sealed box.

This article takes a different route. Instead of treating the assistant as a sealed box, it asks what the smallest useful piece is and how the larger behavior is put together from it. Keep that question in mind as you read, because the answer will shape how you debug, design, and describe these systems.

The course section this article belongs to proposes that you can understand a complex assistant by constructing one yourself, one layer at a time. There are two layers to hold in mind. The bottom layer is a text generator, which is an LLM that can produce written output. The top layer is a set of query techniques, which are ways of directing that generator toward specific jobs. The assistant is what you get when both layers work together.

Reading the diagram from the bottom up gives the order of study. First you learn how a generator produces text, then you learn how to point that generator at specific tasks, and finally you see how those tasks combine into an assistant. Reading it from the top down gives a practical debugging habit. When an assistant behaves unexpectedly, you can ask whether the problem lies in the generator underneath or in the technique that is directing it.

The source material emphasizes two points. The first is that an LLM can generate texts. The second is that an LLM can be coded to generate texts. The second point matters because it turns an abstract capability into something you can build and run yourself. Generation is therefore not only a property you observe in a product; it is a foundation you can construct.

Generated text is not produced all at once. The model writes one token at a time. Once that token exists, it becomes part of the text the model considers next, so the output grows step by step. The model keeps repeating this cycle until a stopping condition is reached. Follow the loop in the diagram below and notice where the text gets longer and where the process ends.

Trace one pass through the loop. The prompt feeds the model. The model writes a token, that token is added to the text, and the stop check runs. If the check says to continue, the model writes another token from the longer text. The loop only ends when the check says the output is finished. This is why a single answer can be many small steps rather than one instant decision.

This part of the course focuses on three jobs: answering questions, summarizing text, and translating text. Each is a distinct query technique. The sections that follow introduce each one and show a scenario where it would apply.

The three techniques share the same generator but differ in what they take in and what they give back. Comparing their inputs and outputs is the fastest way to tell them apart when you are looking at a request.

Put the layers together and follow the data. A user's query enters the system, a query technique determines which job the generator should perform, the generator produces text, and that text comes back out as the response. The diagram shows this path from start to finish.

This flow also explains the debugging habit from earlier. If the response has the wrong shape, for example a full rewrite when a short summary was wanted, check whether the technique was matched to the request. If the text itself is poor, the problem is more likely in the generator beneath it.

Matching a request to its technique is a practical habit. Read the request, look for cues in its wording, and decide which of the three jobs it asks for. Work through the example before trying the practice prompt.