Choosing Representative Examples for Prompts

Imagine you want a language model to sort customer messages into three groups: billing, technical problem, or general question. The simplest approach is a single instruction such as 'Label each message as billing, technical, or general.' Sometimes that works well. Other times the model returns labels in a different style than you wanted, uses a word you did not expect, or treats borderline messages inconsistently.

A natural fix is to show the model a few messages that have already been labeled correctly, placed right before the new message you want labeled. The model then sees the pattern in those examples and continues it. Improving a model's behavior by placing demonstrations inside the prompt is the subject of this article. The key idea is that the examples change what the model produces without any change to the model itself.

The number of examples in the prompt is what distinguishes these cases. A zero-shot prompt has zero examples and relies on the instruction alone. A 1-shot prompt contains one example. A 3-shot prompt contains three. Each example is an input paired with the output that should follow it, so the prompt demonstrates the task rather than only describing it.

A few-shot prompt is usually assembled in a predictable order. It begins with an instruction that describes the task. Next come the examples, one after another, each showing an input and the output that should follow it. The new input you actually want processed comes last, with its output left blank for the model to complete.

The model reads the whole sequence as one block of text and produces a continuation. Because the examples sit inside the same text, the model can compare the new input against them as it writes. Follow the flow in the diagram below, from the components you assemble to the output the model returns.

Notice what does not appear in this flow: any step that updates the model's underlying parameters. The examples travel as text in the request, and the model's behavior on this request is shaped by that text alone.

A model can only read so much text at once. The maximum amount of input it can accept is called its context length. Every example, the instruction, and the new input must fit inside that limit together. The model's output also needs room, so the same space is shared by the answer it will write.

This is why the number of examples you can use is bounded. Adding examples does not stack indefinitely. Once the prompt reaches the model's maximum context length, there is no room for further demonstrations. The diagram below represents a prompt that fills the whole available input. The instruction and examples occupy most of the space, leaving only a small remainder for the new input and the answer.

The two approaches differ in how much work the prompt does before the model begins. A zero-shot prompt is shorter and cheaper to send, but it asks the model to infer the exact style you want from the instruction alone. A few-shot prompt spends more of its length on demonstrations, which tends to narrow the range of acceptable answers. The comparison below shows how the number of examples changes what the prompt contains.

Every token the model processes has a cost, and the input prompt is part of that processing. The source describes this as inference cost: the cost of processing a model's input when it generates a response. Each added example makes the prompt longer, so adding examples also raises inference cost.

This trade-off matters most when the same prompt is sent thousands of times. A modest increase per request can become a large increase in total. Before committing to a design, it helps to estimate how much each added example will cost across all the requests your application will make.

The benefit of examples is not uniform across models. The source reports that for GPT-3, few-shot learning showed significant improvement over zero-shot learning. In the use cases analyzed in Microsoft's 2023 study, however, few-shot learning gave only limited improvement over zero-shot learning on GPT-4 and a few other models. The source interprets this as a sign that more powerful models follow instructions better, so they can perform well with fewer or no examples.

The source also cautions that the study might have underestimated the value of examples in domain-specific work. Its illustration is the Ibis dataframe API. If a model has seen few examples of that API in its training data, including Ibis examples in the prompt can still make a large difference, even for a model that otherwise performs well with zero-shot prompts. The lesson is that a result from one set of use cases does not settle the question for your own application.