Short-term and Long-term Memory in AI Applications

When an AI application runs out of room during a long conversation, a tempting shortcut is to assume that the oldest messages are the least useful. The reasoning is that the conversation has moved on, so the earliest exchanges must be stale, and removing them should cost very little. This belief is common enough that it shapes how many memory systems are built. It is worth examining carefully, because in some conversations it is fatally wrong. The earliest messages can carry the most information, especially when they state what the conversation is for.

Follow what happens to the conversation from the previous example under the simplest possible rule. Each new message enters at the end of the queue. When there is no more room, the message at the front, which is the oldest one, is the first to leave. The diagram traces that sequence. The first message, which sets the goal of the whole task, is the earliest entry, so it is the first to be removed, even though the recent formatting questions are the least important part of the work.

The rule that decides which message leaves is never based on what the message says. It is based only on when the message arrived. That is why FIFO is easy to implement and easy to reason about, and also why it has a predictable blind spot. Because it ignores content, it cannot tell a casual greeting from a sentence that defines the whole task.

Some real systems follow this pattern at the level of the whole conversation. The source notes that API providers such as OpenAI might start removing the beginning of a conversation as it grows. It also notes that frameworks such as LangChain might allow the retention of the last N messages or the last N tokens. In both cases the system keeps the recent part of the conversation and lets the start fall away.

A more complete picture places FIFO inside a larger design with two memories. Short-term memory holds the recent working material. Long-term memory can store the overflow from short-term memory. For any query, the context given to the model contains both its short-term memory and information retrieved from long-term memory. The two memories therefore draw from the same fixed context limit, and that shared budget is what makes overflow a design problem rather than a storage problem.

The source gives a concrete allocation. If 30 percent of the context is reserved for retrieved material, then the model can use at most 70 percent of the context limit for short-term memory. When that threshold is reached, the overflow can be moved to long-term memory. The key lesson is that the two memories share one fixed budget, so reserving more for one leaves less for the other. Increasing the space for retrieved information shrinks the room available for recent conversation, and that in turn changes how much overflow is produced in the first place.

The source describes strategies that avoid dropping information in bulk. Their starting point is human language, which contains redundancy to enhance clarity and to compensate for potential misunderstandings. Repetition and restatement help people, but a memory store pays for every repeated sentence. If redundancy can be detected automatically, the memory footprint shrinks, and fewer items need to be pushed out at all. Two approaches appear in the source: redundancy removal and reflection-based updating.

The three approaches differ in what they consult when they change memory. FIFO consults only arrival order. Redundancy removal consults the content, looking for repeated material and recovering what a summary missed. Reflection consults the newest information and the existing memory at every step, so it can decide to insert, merge, or replace. The diagram groups the decision rule and the outcome of each approach side by side.

Contradictions are a special case of the memory problem. The source gives two common approaches. Some people keep the newer piece of information when two pieces contradict each other. Others ask an AI model to judge which one to keep. The source adds that how to handle a contradiction depends on the use case, so neither approach is correct everywhere. The flowchart turns that advice into a sequence of questions you can ask about your own application.

The source also points out that contradictions are not only a problem. Having contradictions can cause an agent to be confused, but can also help it draw from different perspectives. The design task is therefore not to eliminate every disagreement in memory. It is to decide, for a given use case, which disagreements must be resolved and which can be kept. A single answer is essential when the agent must act on one fact, while several perspectives may be useful when the agent is weighing options.