Memory Retrieval in AI Systems

A model on its own only sees what is placed in front of it right now. That works for a single, self-contained question. It stops working once a conversation grows long, or once a task depends on something said earlier or stored somewhere else. At that point the model needs a way to keep track of information beyond the immediate input.

A memory system is the part of an AI system that handles this problem. It is typically built from two functions. One decides what gets stored. The other pulls stored information back when a task needs it. This article follows the first function, memory management, while keeping the second, retrieval, in view because the two are closely linked.

Memory management is the write side of the memory system. It decides what goes into each store and what comes out. Memory retrieval is the read side. It takes information that already sits in long-term memory and finds the pieces relevant to the task at hand. You can think of management as answering 'what do we keep?' and retrieval as answering 'what do we bring back now?'.

There are two kinds of store that management works on. Short-term memory is bounded by the model's maximum context length. Long-term memory is an external store. Every decision about the contents of either store reduces to one of two operations, which the next section defines.

The definition matters because it limits the vocabulary. Whatever the memory system decides, the decision is an add or a delete, applied to either short-term or long-term memory. Keeping this in mind makes the rest of the topic easier to follow: when you see a memory decision, ask which store it touches and whether it adds or removes something.

The clearest way to understand short-term memory is to follow its contents step by step. Consider a short-term store with room for three items. Each arrival either fits into the free space or forces one existing item to leave. The diagram below traces that process.

Deletion is not always required. If the memory store has enough room for everything that arrives, the system can keep all of it and never run the delete operation. The source states this directly: when storage is limited, deletion might not be necessary. The same reasoning applies whenever capacity is effectively unbounded.

This is the key idea behind the whole topic. Add is always needed when something new should be kept. Delete is needed only when a store has no room left for what arrives. When room is plentiful, a memory system can skip deletion entirely.

Short-term memory is different from the relaxed case above. It is limited by the model's maximum context length, which is a hard ceiling on how much information the model can hold at one time. Because that ceiling cannot simply be extended, short-term memory needs an explicit strategy for what to add and what to delete. Without one, new information has nowhere to go once the limit is reached.

When new information arrives, a strategy evaluates it. That strategy determines whether the item is added to short-term memory, added to long-term memory, or removed. The flow below shows the structure of the choice for a single store. It shows only the shape of the decision. The source does not specify the exact criteria the strategy uses.

Read the flow from the top. The first check is whether the store has room. If it does, the item is added and nothing is removed. If it does not, the strategy must choose something to delete before the add can happen. This is why the same two operations, add and delete, explain every outcome.

Information moves in three directions. It enters short-term memory when it arrives. It can pass from short-term memory into long-term memory as the strategy routes it. And when a task needs something stored earlier, retrieval searches long-term memory and brings only the relevant material back into the model's context. The diagram below shows this path.

Retrieval from long-term memory is similar to RAG retrieval, a pattern in which information is fetched from an external data source and supplied to the model. Long-term memory plays that external-source role here. The system searches for what is relevant to the current task, and only that material is brought forward.

Management decisions affect short-term memory most directly. Its ceiling is fixed, so every arrival pushes the store toward its limit and forces a choice. Long-term memory is less affected because it can grow, so the add operation usually succeeds without a deletion.