Memory Systems for Language Models

A language model answers questions using two sources: what it absorbed during training, and whatever text appears in its current input. Neither source is ideal. Training knowledge goes stale, because the world keeps changing after training ends. The input also has a limited size, so a model cannot simply be handed an entire company's documents and asked a question about them. Retrieval-Augmented Generation, usually shortened to RAG, is a design pattern that gets around both limits. Instead of relying on memory alone, the system first looks up material that might be relevant and then places that material in front of the model.

This article focuses on the first half of that arrangement, the retriever. The quality of a RAG system is tied closely to how well the retriever does its job, so understanding how retrievers decide what is relevant is the key to understanding RAG itself.

A RAG system works in two stages. In the first stage, it retrieves information from an external memory, which is a store of documents, passages, or records that exists outside the model. In the second stage, it passes the retrieved material to the model, which uses it to generate a response. The retrieval stage supplies the facts, and the generation stage turns those facts into an answer.

Tracing the flow step by step: the query enters the retriever, the retriever searches the external memory and selects material that might be relevant, that material is placed in the model's input, and the model generates the answer from it. The stated expectation of this design is that the answer will be more accurate than what the model could produce from its own memory, because the answer is anchored to material the system actually found.

A term-based retriever decides relevance by looking at the words themselves. A query is broken into its terms, and the retriever favors documents that contain those same terms. Documents where the matches are strong receive more weight. Systems such as Elasticsearch and the BM25 ranking method are examples of this approach.

The main practical advantage of term-based retrievers is that they are much lighter to set up and run. They often give a solid starting point against which more advanced retrievers can be measured.

An embedding-based retriever takes a different route. Rather than comparing words directly, it relies on vector search, in which items are represented as vectors and the system looks for the ones closest to the query's vector. Because the comparison happens in this representation space, the approach has the potential to recognize relevance even when the wording differs. This is why embedding-based retrievers can outperform term-based algorithms.

Vector search is not a technique invented only for language models. It is also the backbone of many core internet applications, including search and recommender systems. Many vector search algorithms developed for those applications can be reused in RAG, so the retrieval side of a RAG system can draw on a large body of existing work.

The two families differ along several dimensions at once: what they match on, how much effort they require, and where their advantages appear. Placing them side by side makes the choice easier to reason about than reading about each separately.

RAG can be understood as a special case of an agent in which the retriever is one of the tools the model is allowed to use. Both RAG and agentic designs let a model get past its context limit and stay current, because both bring in outside information. The agentic pattern goes further. An agent is defined by its environment and by the tools it can access. Inside an AI-powered agent, the model acts as the planner: it analyzes the task, considers several possible solutions, and selects the most promising one.

Planning matters because a complex task may need many steps, and a model must be powerful enough to plan them. The planning ability can be strengthened with reflection and with a memory system that keeps track of progress. Adding tools also adds capability: the more tools a model is given, the more capabilities it has, which lets it take on harder tasks.

RAG and agents share one more feature. Both are prompt-based methods. They shape the model's output solely through what goes into its input, without changing the model's internal weights. That is what separates them from approaches that modify the underlying model itself.