Tools and Function Calling for Agents

Suppose you are handed an AI agent and asked what it is able to do. Two questions come up almost immediately. The first is where it operates: what world does it perceive and affect? The second is what it can actually do inside that world. These two questions are linked. The world an agent works in shapes the things it could possibly do, and the things it is equipped to do shape how much of that world it can reach.

An agent's environment is defined by its use case. The easiest way to see this is to ask, for each agent, what it is supposed to act upon. An agent built to play a game has that game as its environment. An agent that scrapes documents from the internet has the internet as its environment. A cooking robot works in a kitchen. A self-driving car works in the road system and the areas beside it.

Once the environment is fixed, it determines which tools an agent could potentially use. A chess game is a useful case. The only possible actions for an agent inside it are the legal chess moves. Nothing outside the rules of the game is available to it. The environment, in other words, sets the boundary of what actions even make sense.

The relationship runs in both directions. The tool inventory an agent carries restricts which environments it can actually work in. Consider a robot whose only action is swimming. It is confined to a water environment, because on land it has nothing it can do. The tools fix the reachable world just as the world fixes the usable tools.

Tools also expand what an agent can do. A model that only produces text has a small set of possible actions, and each tool adds a new one. Many generative AI applications people use every day are agents with access to tools, although many of those tools are simple. ChatGPT can search the web, run Python code, and generate images. Each of these capabilities is an action that is unavailable to a model with no tools at all.

Retrieval-augmented generation (RAG) systems follow the same pattern. A RAG system can be viewed as an agent whose tools are a text retriever, an image retriever, and a SQL executor. Each tool is a specific capability the agent can call on.

A coding agent called SWE-agent, built on top of GPT-4, shows both ideas together. Its environment is a computer with a terminal and a file system. Its set of actions includes navigating a repository, searching files, viewing files, and editing lines. The environment supplies the places where those actions apply, and the actions are what let the agent change that environment.

An agent is typically given a task by a user. It does not simply return one answer. It processes the task together with feedback from its environment, plans a sequence of actions, and decides when the task is finished. The diagram below shows the path a single tool call takes. The agent's decision goes out to a tool, the tool acts on the environment, and the result comes back to the agent so that it can make its next decision.

The key point in the diagram is the return arrow. The agent goes back to reasoning after every tool call. Its next move depends on what it just learned, not only on the original request.

To see the loop in action, take a RAG system working with tabular sales data. It has three actions: generating a response, generating a SQL query, and executing a SQL query. Given the request to project sales revenue for a product over the next three months, the agent carries out a sequence of moves. Each move is a choice based on what the agent has just seen.

The sales run described in the source had seven reasoning-and-action moments. Each one is a place where the agent can go wrong. A query can be malformed, a result can be misread, or a plan can be poor. This is the reason the next part of the article matters.

Two details in the sales run are worth noticing. The first is that the agent judges its own progress. It decides when the information is enough and when the task is complete. The second is that this judgment is part of the loop, not an afterthought. The source describes this as the AI component deciding whether the task has been accomplished, which is what allows the agent to stop.

Because an agent needs many steps, its overall accuracy depends on how the individual steps combine, not only on how good any single step is. This is called the problem of compound mistakes. The diagram below shows the chain. Each step must succeed for the run to stay on track, so the chance of a fully correct run shrinks as the chain grows.

The source gives a concrete case. If a model is 95% accurate per step, then over 10 steps the overall accuracy drops to about 60%, and over 100 steps it falls to about 0.6%. The table below lays out those figures so you can see how quickly the trend bends downward.

The pattern matches multiplying the 95% figure by itself once for each step: $0.95^{10} \approx 0.60$ and $0.95^{100} \approx 0.006$. A small per-step error rate, repeated over many steps, adds up to a large chance that something has gone wrong somewhere along the way.

Compared with non-agent uses of a model, agents typically require more powerful models. The source gives two reasons. The first is the compound-mistake effect just described. The second is that agents with access to tools can perform more impactful tasks, but any failure could carry more severe consequences. A text answer that is wrong can be ignored. An action that is wrong may have already changed something in the environment.

Cost works in the other direction too. A task that requires many steps takes time and money to run. The source argues that this cost can still be worthwhile, because autonomous agents can save a great deal of human time. Whether an agent is worth deploying depends on weighing that saving against the expense of running many steps and the damage a failure could cause.