Context and Conversation History in Models

It is common to talk about a model as if everything it knows sits inside it. In practice, the source describes three distinct memory mechanisms that a model can rely on, and each one differs in where its information comes from, how long it lasts, and how quickly it can be used. Understanding these differences helps you reason about why a model answers well in one moment, forgets something in another, or can be corrected without retraining.

A model has three memory mechanisms. The first is internal knowledge, which the model retains from the data it was trained on. The second is the model's context, which holds previous messages in a conversation so the model can use them when it generates later responses. The third is a set of external data sources that the model can reach through retrieval, such as a RAG system. Each mechanism holds different kinds of information, and each reaches the model by a different route.

Read the three columns as three different answers to the same question: where does the model get the information it is using right now? Internal knowledge is already present when a query begins. Context is assembled from the conversation in front of the model. Long-term memory is fetched on demand. The difference in route is what makes each mechanism behave the way it does.

Two properties define internal knowledge. First, it does not change unless the model itself is updated. Second, the model can access it in every query, because it is already inside the model. This makes internal knowledge the most stable of the three mechanisms. It is also the least flexible: to change what is stored there, the model has to be changed. The source notes that information essential for all tasks belongs here, placed through training or finetuning.

The second mechanism is the model's context. Previous messages in a conversation can be placed into the context, which lets the model use them when it generates later responses. Because the context is what the model is working with right now, the source treats it as the model's short-term memory.

Short-term memory has two defining trade-offs. It is fast to access, which makes it convenient for the current task. But its capacity is limited, and it does not persist across tasks. Each query does not carry it forward on its own. For this reason, the source says context is often used to hold the information that matters most for the current task, and not everything the model might ever need.

The third mechanism is made of external data sources that the model can access through retrieval. The source gives a RAG system as an example. These sources sit outside the model, so they can be persisted across tasks. This is why the source calls them the model's long-term memory.

The most useful contrast with internal knowledge is this: information in long-term memory can be deleted without updating the model. Internal knowledge is baked in, so removing it requires changing the model. Long-term memory is separate, so the information can be changed or removed at the level of the data source. The source offers a human parallel: your long-term memory is augmented with books, computers, and notes.

Retrieval is the route by which this information reaches the model. It is not already inside the model when a query begins. It is fetched when it is needed, and the fetched material is then available to the model while it answers. The diagram below traces that path.

The three mechanisms are easier to understand when you follow a single piece of information through an interaction. The scenario below is an original illustration, built to show how the mechanisms differ. Read it first and notice where each fact ends up before checking the explanation.

The source gives a clear rule for deciding where information should live: the choice depends on how frequently the information is used. Information that is essential for all tasks belongs in internal knowledge, built in through training or finetuning. Information that is rarely needed should reside in long-term memory. Short-term memory is reserved for immediate, context-specific information. Follow the decision path below to place a piece of information.

Choosing between keeping information in context and storing it in an external source is a trade-off. Context is fast to use but limited in capacity and does not persist across tasks. External sources persist across tasks and can be changed without updating the model, but the information has to be fetched through retrieval when it is needed. The table below compares the two.