Query Caching Criteria

Imagine an AI application that receives the same kind of question many times a minute. Each time a question arrives, the application may call a model, run a multi-step reasoning process, search a vector index, or query a database. Some of these actions are slow, and some cost money to run. If the application has already produced an answer for a request, running the whole process again is wasted effort.

Exact caching is a way to avoid that waste. The system keeps a record of past results and reuses them when the same request comes back. This article follows one question from start to finish: when the system receives a request, how does it decide whether a stored result can be used, and what has to be true before that reuse is safe? The answer depends on the word exact, so we start there.

The word exact defines what counts as a hit. A cache hit happens when an incoming request matches a stored key. A cache miss happens when no stored entry matches, and the system must do the work itself. In the product example used by the source, the request is a request to summarize one particular product. Before doing any work, the system checks whether a summary of this exact product already exists. If it does, the stored summary is fetched. If it does not, the system creates the summary and stores it for next time.

The lookup key is the request itself, and only an identical key produces a hit. A summary of one product cannot be handed back for a different product, even if the two products sound alike. The same pattern applies to embedding-based retrieval. When an incoming query is already in the vector search cache, the stored result is fetched. If it is not, a vector search runs for that query and its result is cached. In both cases the purpose is to avoid redundant work: repeating a search or a summary that has already been done for the same request.

Think of the cache as a table of keys, where each key points to a stored value. The system turns an incoming request into a key, looks that key up, and either finds a value or does not. Because the match is exact, even a tiny difference in the key produces a different key and therefore a miss.

In this picture, the stored entry is keyed to product A100. An incoming request for product A101 produces a different key, so the lookup finds nothing and the request is a miss, even though the two products are almost identical. The same logic would reject a summary for a request that differed only in spacing or wording, because the key, not the meaning a person would assign to it, is what the system compares.

The best way to understand exact caching is to watch the cache change. Below is a short run of four requests to summarize products. The cache starts empty. For each request, the system first checks whether a summary of this exact product already exists. Study the timeline, then notice what changes after each step.

An exact cache can be implemented with in-memory storage, which allows fast retrieval. The source notes a limit to this choice: in-memory storage is restricted in how much it can hold. Because of that limit, a cache can instead be built on a database such as PostgreSQL or Redis, or on tiered storage, which is used to balance speed against storage capacity.

A cache that only ever grows will eventually become a problem, especially when storage is limited. The source states that an eviction policy is crucial for managing cache size and maintaining performance. An eviction policy is the rule the system uses to decide which entries to remove when space is needed.

The source names three common eviction policies. Least Recently Used (LRU) looks at how recently each entry was used. Least Frequently Used (LFU) looks at how often each entry was used. First in, first out (FIFO) looks at the order in which entries arrived. The source lists these names without describing their internal mechanics, so the names themselves are the core facts to remember.

Not every request is worth storing. The source says how long a query should stay in the cache depends on how likely that query is to be called again. Two categories are called out as poor candidates.

The first category is user-specific queries, such as a question about the status of one's recent order. Such a result is unlikely to be reused by other users, so it should not be cached. The second category is time-sensitive queries, such as asking about the weather. A stored answer to a question whose correct answer changes over time becomes misleading, so caching it makes less sense.

The source also notes that many teams train a classifier to predict whether a query should be cached. The source does not describe how such a classifier is built, so the important idea is the goal: predicting the likelihood that a query will be reused.

Read the flowchart as a set of questions asked in order. The first two checks come from the source: user-specific and time-sensitive queries are poor candidates. The hidden-data check follows from the leak example in the next section. The last check reflects the source's point that caching is most valuable when a request involves multiple steps or time-consuming actions such as retrieval, SQL execution, or web search.

Caching has a hazard that goes beyond wasted storage. The source warns that caching, when not properly handled, can cause data leaks. The trouble comes from a request that looks generic but actually depends on who is asking.

Consider an ecommerce site. A user asks, 'What is the return policy for electronics products?' The question sounds the same for everyone, but in this case the return policy depends on the user's membership. To answer, the system first retrieves that user's information, then generates a response that contains it. If that response is stored under a key built only from the question text, the key does not include the user's identity.

Follow the messages in order. User A's question creates an entry. User B then asks the same question. Because the key matches exactly, the cache reports a hit and returns A's answer, which reflects A's membership. Nothing in the key was wrong by the exact-match rule; the leak comes from the key leaving out the one fact that changed the answer. A safe design either keeps such results out of the shared cache or includes the user's identity in the key so that a hit can only occur for the same user.