Controlling Prompt Length

Many prompts teach a model a task by showing it a few finished examples before asking for the one result you actually want. In this lesson the task is sorting items into two categories. Each example pairs an item with its label, and the last line leaves the label blank. The prompt below is the whole input. Read it as plain text first, noticing how the pattern is built before any explanation is attached to it.

Before you read on, make a prediction. Look only at the finished pairs and ask what relationship they set up between an item and its label. Write down a one-word guess for what belongs after the last arrow, and one sentence describing the rule you think the pairs follow. Keep that guess. You will test it as you work through the rest of the article, because the reasoning you just did is close to what the model does when it reads this block of text.

A model does not read a prompt the way a person reads a sentence. It reads a sequence of tokens, and the length of that sequence is the input token length. The finished pairs, the open slot, and every piece of formatting between them all become part of one block of input that the model takes in before it produces any output. Nothing in the block is a separate message. The examples and the unfinished line arrive together as a single sequence.

Once you see the prompt as one block, the question changes. You are no longer asking which part the model reads most carefully. You are asking how long the whole block is and which exact position is the one the model is expected to complete. Those two questions drive the rest of this article.

A few-shot prompt has two kinds of content. The finished pairs are fixed examples that set the pattern. The open slot is the final position, where the item is present but the label is missing. The model is asked to complete that slot. In the sorting prompt, the fixed examples are apple and rock, plus bread, and the open slot is the blank after the arrow on the pebble line.

The diagram shows the order of the pieces, not the inner workings of the model. Each finished pair sits on the same path as the open line. The pairs supply the pattern, and the open line asks for the answer. This is why a completion that appears after the arrow on the last line is the thing being tested.

Every finished pair you include adds to the input token length, because every pair is part of the block the model must read before it reaches the open slot. Changing the example format changes the first stage of this path, so it can change how many tokens the whole input occupies. If you are concerned about how long the input becomes, the practical rule is to choose an example format that uses fewer tokens.

Use the shorter format as the default when input length is a real constraint and the shorter format matches the longer one on your task. Keep the longer format when it is needed to get the completions right. The decision depends on the trade-off between token cost and how well the completions match what you want.

The pattern in the examples tells the model what kind of answer to give, but the output format should be stated explicitly. An open slot is only useful when the model knows what shape its completion should take. In the sorting prompt, the completion is a label from the set the finished pairs established, either edible or inedible, written after the arrow. Specifying the output format means making that expectation visible, so the completion lands in the slot you intended rather than being inferred from the examples alone.

When a completion is not what you expected, resist the urge to rewrite the whole prompt at once. Walk the path from the finished pairs to the open slot and check each stage in order. Start with the finished pairs and confirm that each one follows the same arrow-and-label shape. Next check the open slot and confirm that it ends in the same shape with nothing after the arrow. Then compare the completion's form with the labels the pairs established. The first stage where the prompt stops matching your intention is usually where the divergence begins.

Now return to the guess you wrote at the start. Does the rule you described match the pattern the finished pairs establish? Compare your one-word guess with the label the pairs point toward, and note which stage of the path you relied on to decide.