Prompt Caching

Imagine a customer support assistant built on a language model. Every time a customer sends a question, the application sends the model a fresh request. Most of that request is the same: a long block of instructions describing the company's tone, policies, and formatting rules. Only the final line, the customer's actual question, is new. The model reads the entire request every time, including the instructions it has already read thousands of times that day. That repetition is the problem prompt caching is designed to address.

The same pattern shows up in other applications. Suppose you have built an application that classifies customer messages, and you have found that the model performs well when you show it a handful of labeled examples before each new message. The examples work, but they are not free. Every time the application sends a request, those examples travel with it. This article follows one question from that situation: what does it cost to keep sending the same material, and what can be done about the repeated work?

A prompt is the full text the model receives on each call. When you teach a model by example, the examples become part of that text. Each example adds input tokens, and the model must process all of them every time. The source identifies two direct consequences: latency goes up, and cost goes up. Because the examples are repeated in every prompt, the overhead is paid again on every request, not once.

The effect is easiest to see by comparing the same request with and without the examples. The diagram below shows the per-request input in both arrangements. In the first, the labeled examples sit inside every prompt along with the instructions and the customer message. In the second, the examples have been moved out of the prompt, so each request carries only the instructions and the message.

The key point is that the overhead is per request. A single prompt with a few examples may look harmless, but the examples are sent again with every customer message. The cost of teaching by example therefore scales with the number of calls, not with the number of examples you wrote.

Finetuning offers a different route. Instead of placing the examples in each prompt, you finetune a model on those examples. The examples are then used during training rather than being resent with every request. The practical result is that the prompts sent to the finetuned model can be shorter, because they no longer need to carry the examples along.

This removes the repeated input tokens that the examples would otherwise add. The question of when that payoff is worth it depends on how the token savings compare with the alternatives, and one important alternative is prompt caching.

The most common overlapping segment is the system prompt, the instruction block that frames the model's behavior. Without a prompt cache, the model processes the system prompt along with every query, so a system prompt that is identical across a thousand queries is handled a thousand times. With a prompt cache, the system prompt is processed for the first query, and later queries can reuse that processed result rather than starting over.

This is why overlapping text is the reason the cache helps. The cache has nothing to offer for text that is unique to each request. It helps only when the same leading text reappears, and the source notes that it helps most when that shared material is long. Shorter repeated text saves less per request, while long repeated text is expensive to process every time.

Follow the support assistant through its first two requests. The first request contains the full system prompt followed by a customer question. The second request contains the same system prompt followed by a different question. The system prompt is identical in both, so it is the overlapping segment. The customer questions differ, so they are new text on each request.

Read the diagram from top to bottom. The first request processes the system prompt in full, and the processed result is stored. The second request's system prompt matches the stored segment, so its processed result is reused rather than rebuilt. Only the customer question, which is new text, still goes through full processing. The work skipped on the second request is the work on the shared leading text.

The source notes that finetuning helped optimize token usage before prompt caching was introduced. Prompt caching allows repetitive segments of a prompt to be cached and reused. Because the repeated examples can now be reused rather than reprocessed in full on each call, the token-saving argument for finetuning is much weaker than it once was.

The table shows why caching narrows the gap. Finetuning removes the examples from the prompt, and caching removes most of the repeated processing of those examples. The advantage that remains for finetuning is not about token usage in the simple sense, which is where caching has reduced the difference.

Prompting has a ceiling that finetuning does not share in the same way. The number of examples you can place in a prompt is still limited by the maximum context length of the model. Even with caching, you cannot keep adding examples beyond what fits in the context window. With finetuning, the source states there is no limit to how many examples you can use. This is the lasting advantage that remains after caching has been taken into account.

In the left column, the prompt holds the instructions and as many examples as the context window allows. The final example is the one that may not fit, which is the ceiling. In the right column, the examples have been absorbed into the model during finetuning, so the prompt holds only the customer message. The source's statement is that the number of examples in finetuning is not limited in this way.

Finetuning is not a shortcut around careful work. The source makes the point that both finetuning and prompting experiments require systematic processes. Prompt experiments are valuable in their own right, and they also prepare the ground for finetuning. Working through prompts helps developers build an evaluation pipeline, write data annotation guidelines, and set up experiment tracking practices. Each of these becomes a stepping stone when you later decide to finetune.

For the support assistant, this means that testing prompt variations is not wasted effort even if you later finetune. The evaluation method and labeling guidelines you develop while prompting are the same tools you will need to build a finetuning dataset.

The source gives a concrete scale for the benefit. Suppose the system prompt is 1,000 tokens long and the application makes one million model API calls per day, with every call sending that same system prompt. Without caching, the model processes the system prompt on every call. The daily total of repeated input tokens is the system prompt length multiplied by the number of calls.

The total grows linearly with traffic. If the same prefix is sent half as often, the daily repeated total halves, and if the prefix is shorter, the total shrinks in proportion. A long shared system prompt on a busy endpoint is the case where caching matters most.

Caching is not free, and the source is direct about the trade-offs. A prompt cache takes up memory, and as with the KV cache, its size can become large. That memory has to be paid for in some form, whether through the infrastructure that holds it or through fees charged by a provider.

The source reports figures from two model API providers. Google Gemini offers prompt caching in which cached input tokens are priced at a 75 percent discount compared with regular input tokens. Anthropic's prompt caching is described as offering up to 90 percent cost savings and up to 75 percent latency reduction. The source notes that the savings grow as the cached context gets longer.