Cache Implementation

Imagine an AI application that receives the same kind of question many times a minute. Each time, it may call a model, run a multi-step reasoning process, search a vector index, or query a database. Some of these actions are slow, and some cost money to run. If the application has already produced an answer for a request, running the whole process again is wasted effort. Exact caching avoids that waste by keeping a record of past results and reusing them when the same request comes back.

This article follows one question from start to finish: when the system receives a request, how does it decide whether a stored result can be used, and what has to be true before that reuse is safe? The answer depends on the word exact, so we start there.

The source describes a concrete case: a user asks a model to summarize a product. Before doing any work, the system checks whether a summary of this exact product already exists. If it does, the stored summary is fetched, which is a hit. If it does not, the system creates the summary and stores it for next time, which is a miss. The best way to understand this logic is to watch the contents of the cache change across a short run of requests.

Read the diagram as a lookup table. The request Alpha finds its stored value because the key matches exactly. The request Alpha Pro Max looks close to the stored key Alpha Pro, but it is not identical, so the cache does not return that entry. A near match is still a miss, because exact caching never reuses a result for a request that is merely similar.

The same pattern appears in embedding-based retrieval, which the source also names as a use of exact caching. When an incoming query is already in the vector search cache, the stored result is fetched. If it is not, a vector search runs for that query and its result is cached. In both cases the purpose is to avoid redundant work: repeating a search that has already been done for the same query.

Exact caching is most appealing for certain requests. The source highlights queries that involve multiple steps, such as chain-of-thought reasoning, and time-consuming actions such as retrieval, SQL execution, or web search. The more work a request requires, the more there is to save by reusing its answer.

An exact cache can be implemented with in-memory storage, which allows fast retrieval. The source notes a limit to this choice: in-memory storage is restricted in how much it can hold. Because of that limit, a cache can instead be built on a database such as PostgreSQL or Redis, or on tiered storage, which is used to balance speed against storage capacity.

Reading this as a tradeoff helps. Faster storage tends to hold less, and larger storage tends to be slower to read. The cache design is a decision about where on that spectrum your results should live, given the volume of requests your application handles. A small volume of requests may fit comfortably in fast memory, while a high volume may push the design toward larger storage.

A cache that only ever grows will eventually cause a problem, especially when storage is limited. The source states that having an eviction policy is crucial for managing cache size and maintaining performance. An eviction policy is the rule the system uses to decide which entries to remove when space is needed.

The source names three common eviction policies. Least Recently Used (LRU) is one. Least Frequently Used (LFU) is another. First in, first out (FIFO) is the third. The names indicate what each policy looks at: LRU is concerned with how recently an entry was used, LFU with how often it was used, and FIFO with the order in which entries arrived. The source lists these names without describing their internal mechanics, so treat the names and what they track as the core facts to remember.

Not every request is worth storing. The source says how long a query should stay in the cache depends on how likely that query is to be called again. Two categories are called out as poor candidates. The first is user-specific queries, such as a question about the status of one's recent order. That result is unlikely to be reused by other users, so it should not be cached. The second is time-sensitive queries, such as asking about the weather. A stored answer to a question whose correct answer changes over time becomes misleading, so caching it makes less sense.

The flowchart turns the source's guidance into a sequence of questions. The first two questions screen out requests that are user-specific or time-sensitive. The third asks whether the request is likely to be called again, which the source describes as the factor that governs how long a query should stay in the cache. Only requests that pass all three checks reach the exact-key lookup. The source also notes that many teams train a classifier to predict whether a query should be cached. The source does not describe how such a classifier is built, so the important idea is the goal it serves: predicting the likelihood that a query will be reused.

Caching has a hazard beyond wasted storage. The source warns that caching, when not properly handled, can cause data leaks. Consider an ecommerce site where a user asks, 'What is the return policy for electronics products?' The question sounds the same for everyone, but in this case the return policy depends on the user's membership. To answer, the system first retrieves that user's information, then generates a response that contains it. If that response were cached under the generic question and then returned to a different user with the same question, the second user would receive an answer built from someone else's membership details.