Tool Selection for AI Agents

Picture a model working entirely on its own. A language model can produce text, and an image generator can produce pictures. Each one is good at a single kind of output. Nothing about either model lets it check the weather, look up a bank balance, or send a message. Its reach ends where its own inputs and outputs end.

An agent is different because it can be connected to things outside the model. A system does not need external tools to count as an agent, but without them its capabilities stay narrow. The question this article follows is simple: once an agent can reach beyond the model, what kinds of tools does it need, and what does each kind change about the agent's behavior?

Tools do two jobs. Some let an agent take in information about its surroundings, and others let it change something in those surroundings. Both jobs matter. An agent that can only observe is stuck watching, and an agent that can only act has no picture of what it is acting on.

Before reading further, compare two situations. In the first, a model receives a request and can only answer in the form it was built for. In the second, the same model sits inside an agent that can call tools. The diagram below shows how the reach of the system changes. Notice that the model is the same in both columns; what differs is the set of tools attached to it.

In the standalone case, the model's reach ends where its inputs and outputs end. In the tool-equipped case, the model can recognize that a task needs something it cannot do well alone and pass that part to a tool. The reach of the agent is therefore the combined reach of the model and every tool it can call, not the reach of the model by itself.

Every tool falls into one of two directions. A read-only action lets the agent gather information from a data source without changing anything. A write action lets the agent change its environment or the data sources it depends on. Write actions carry greater risk, because their effects persist after the tool call has finished.

The difference is easiest to see when one tool can do both. A SQL executor can retrieve a data table, which is a read. The same executor can also change or delete that table, which is a write. An email API can read a message and can also send a reply. A banking API can report a current balance and can also initiate a transfer. The tool is the same kind of system in each case; only the operation determines whether the action is a read or a write.

The direction of data flow is the key idea. A read flows from the data source back to the agent, so the agent gains information without altering anything. A write flows from the agent outward into the environment, so the agent's decision now has consequences. Write actions are what make automation possible. An agent could research prospective customers, draft and send emails, read the replies, follow up, record orders, and update a database, all without a person handling each step.

Once an agent's inventory is in view, its tools can be sorted into three families. The first is knowledge augmentation, sometimes called context construction, where tools enrich what the agent knows at the moment it is working. The second is capability extension, where tools cover tasks a model handles poorly by itself. The third is environment interaction, the family of tools that let the agent act upon its environment. That third family corresponds to the write actions described in the previous section.

Knowledge augmentation addresses a limit on what a model knows. A model's knowledge is fixed at the point its training ends, and its working context is limited to what is in front of it. Knowledge augmentation tools supply additional material into that working context, so the agent does not have to memorize everything in advance. The Chameleon agent, which combines a language model with a set of tools, uses knowledge retrieval in this way: it adds retrieved material to what the model has to work with.

Capability extension addresses weaknesses inside the model. Math is the standard illustration, because models are notoriously weak at arithmetic. Rather than spending large amounts of training effort to make a model good at arithmetic, it is far more resource-efficient to give the model access to a calculator tool. Simpler tools in this family include a calendar, a timezone converter, a unit converter, and a translator for languages the model handles poorly. In each case the tool does a narrow job reliably, and the model does the reasoning around it. A code interpreter is a more powerful member: it runs a piece of code, returns the output, or analyzes why the code failed.

Tools only help if the agent knows which ones to use and in what order. That job belongs to the planner. The model is responsible for solving a task, and a task is defined by its goal and its constraints. The goal says what must be achieved, and the constraints limit how it may be achieved. The planner reads the incoming request and forms a plan that respects both.

Complex tasks cannot be handled in a single leap, so the agent produces a plan, which is a roadmap of the steps needed to accomplish the task. Producing a good plan usually means understanding the task, considering several ways to achieve it, and choosing the most promising one. In the multimodal case, the planner's first decision is which output types the request needs. The diagram below traces that routing.

The routing works in two ways. In the diagram, the planner decides whether text generation, image generation, or both are needed. Either branch can run alone. When the request calls for both, their outputs are combined into one reply. The same routing pattern explains how ChatGPT produces text and images, since it relies on DALL-E as its image generator.

The idea also runs in the other direction. A model that accepts only text can call an image captioning tool to interpret pictures, a transcription tool to handle audio, or an OCR tool to read text out of PDFs. The planner decides which of these tools to call before the text model reasons over their results. The model never learns to read the picture; it learns when a picture needs reading and passes the work to the right tool.