Tool Use in Language Models

Suppose you ask a language model a question about your company's internal leave policy. The model was trained on a large body of text, but that training did not include your company's private handbook, and it certainly did not include last month's revision. The model has no direct access to the document, so any answer it gives about the policy is at best a guess.

A natural fix is to paste the handbook into the prompt. That does not always work, because a model can only read a limited amount of text at once, and a long document may not fit. Two families of techniques address this kind of gap. Both work the same way at a high level: they change what the model sees at the moment it answers, rather than changing the model itself. This article follows that idea from its simplest form, retrieval-augmented generation, to the broader idea of an agent.

Retrieval-augmented generation, usually called RAG, is defined as retrieving relevant information from external memory first, and then using that information to generate more accurate responses. The word order in that definition matters. Retrieval comes first, and generation depends on its output.

Step one is retrieval. The user's question is passed to a retriever, which searches an external memory. The external memory can be any store of documents that the system can search, which is why RAG lets a model draw on material that was never part of its training. The retriever returns the pieces of information most likely to be relevant to the question.

Step two is generation. The retrieved pieces are handed to the language model, and the model writes its answer from those pieces. The model is not asked to remember everything. It is asked to write well from the material it has been given.

Because generation depends on retrieval, the retriever decides which information the model ever sees. If the retriever returns the wrong passages, the model is writing from the wrong material. If it returns the right passages, the model has what it needs to answer well. This is why the source states that retriever quality determines how well a RAG system performs.

The source identifies two broad kinds of retriever, term-based and embedding-based, and notes that they trade lightness against potential effectiveness. Choosing between them is a design decision about this trade-off. The important point for a learner is that the retriever is not a minor detail. It is the gate through which all of the model's evidence must pass.

RAG is one pattern. An agent is a broader pattern, and the source defines it by two things: its environment and the tools it can access. In an AI-powered agent, the AI acts as the planner. The planner analyzes the task it has been given, considers different solutions, and picks the most promising one.

Read the diagram as a containment picture. The agent is defined by its environment and its tools, and the planner is the part that decides what to do with them. The retriever from the RAG example appears here as one tool among others. Nothing in the agent's structure requires the retriever to be the only tool, or even the most important one.

Some tasks cannot be solved in one move. The source notes that a complex task can require many steps, and solving it therefore requires a powerful model to plan. Solving such a task means the planner must repeatedly pick a tool, let that tool act on the environment, and then decide what to do next, while keeping track of where the task stands.

Walk the cycle with a simple made-up task: an agent is asked to prepare a summary of several documents. The planner first picks the retriever tool to locate the documents. The retriever acts on the environment by searching the store. The planner then reviews progress and sees that only some documents are gathered, so the task is not complete. It picks the next step and continues until the check says the task is complete. The loop repeats until the planner decides it is finished.

The source also explains why the planner must keep track of its position. A model's ability to plan can be strengthened with reflection, and with a memory system that helps it keep track of its progress. Both support the same goal: the agent must remember where it is in a multi-step task and what it has already tried.

Memory matters for a second reason too. Both RAG and agents work with a lot of information, and that information often exceeds the maximum context length of the underlying model. A memory system is what lets the model manage and use all of the information it has, including material that cannot fit into a single prompt at once.

RAG and agents are closely related. The source describes the RAG pattern as a special case of an agent, one in which the retriever is the tool. Both patterns are prompt-based. Seen this way, RAG is the agent idea narrowed to a single kind of tool, used for a single purpose, which is searching external memory.

The comparison shows what RAG gives up. A full agent has a planner that analyzes the task and chooses among solutions, and it can draw on many tools. A RAG system has one tool, the retriever, and follows the fixed two-step pattern of retrieving and then generating. The narrower design is easier to understand and control, but it does not have the flexibility of an agent that picks its own path.

A natural instinct is to give an agent more tools. The source supports part of that instinct: the more tools a model is given, the more capabilities it has, which lets it solve more challenging tasks. Each tool widens what the agent can accomplish.

The same source warns about the other side of the trade. The more automated an agent becomes, the more catastrophic its failures can be. Each added tool and each step taken without a person checking it widens the range of things that can go wrong. Extra capability and extra risk arrive together, because both come from the same freedom to act without stopping.