Tool Use in AI Agents

Imagine an agent that is asked to plan a task, carry it out using a set of tools, and report back. The answer it gives looks fluent and confident. How would you know whether the agent actually did its job well? A fluent answer does not settle the question, because the report can sound right even when one of the steps behind it went wrong.

The starting point for answering that question is not a score on a leaderboard. It is a list of the specific ways the agent can go wrong. In this sense, evaluation is the work of detecting failures. Before you can measure how well an agent performs, you need to know what failure looks like for it.

This article follows that approach. It names the main categories of agent failure, shows how one planning error is separated from another, and then traces a short plan call by call to find the point where it would break.

The more complex a task an agent performs, the more places there are where something can break. A single model answering one question has relatively few points of failure. An agent, by contrast, decides which steps to take, chooses tools, passes arguments to those tools, and works through many actions in sequence. Each of those decisions is a place where the run can go wrong.

This is why agents need their own treatment in evaluation rather than being judged only by the standards that apply to any AI application. Some failures are shared with every AI application. Agents also have failures that come from three sources: planning, tool execution, and efficiency.

The three categories are siblings. Each one can be studied and measured on its own. The diagram arranges them as a family tree under the broader heading of agent failure modes. Reading it from the top down shows where planning failures sit, and how the tool use failures that the next sections describe are nested beneath planning.

The source gives concrete cases only for planning failures, so this article concentrates there. Tool execution and efficiency are named as categories but are not worked through. The structural point still carries forward: an agent can fail while deciding what to do, while carrying out a chosen action, or while doing either in a wasteful way. Each category needs its own measurement.

Under that second heading the source identifies two distinct errors. A valid tool may be called with invalid parameters, meaning the structure of the call is wrong. Or a valid tool may be called with incorrect parameter values, meaning the structure is right but the content is wrong. Together with the missing-tool error, that gives three specific ways of misusing tools.

Each of the three is a distinct mistake that can appear in a plan. They are worth separating because they are caught in different ways, and because fixing one does not automatically fix the others.

The three planning errors are easier to recognize when the calls are placed side by side. The example below uses a single inventory entry for every call. The inventory defines one tool, set_reminder, which takes one integer parameter called minutes. Read each planned call against that inventory line before deciding whether the call is sound.

The two parameter-related errors are easy to confuse because both involve a valid tool called with bad arguments. The distinction is about what is wrong. In a structural error, the shape of the call does not match the tool's required parameters: there are too many parameters or the wrong kind of parameter. In a value error, the shape is right and only the content is wrong.

The practical lesson is that a structural error can often be caught mechanically, because the call either matches the tool's required parameters or it does not. A value error is harder to see from the call alone. A value of 100 is perfectly legal as a number; it is only wrong relative to what the task demanded. This is why value errors have to be measured against a reference of the correct answer, not simply checked for form.

Identifying error types is only part of the work. In practice you often need to find where in a run the plan would break. The sequence below follows a plan for the task remind me in 10 minutes to stretch, and then check my calendar. The plan makes three calls. The agent checks each call against the inventory before running it.

Read the sequence as a series of ordered messages. Before call 1, the state holds only the plan. After call 1 is confirmed and run, the reminder exists and the agent has one successful result. Before call 2, the agent looks up fetch_calendar. The inventory has no such entry, so the run breaks at step 6, which is a tool use failure of the first kind. The third planned call, set_reminder(minutes=100), is never reached. Even if it had been reached, it would have been a value error, because the number does not match the 10 minutes the task asked for.

Identifying failure modes is only half of evaluation. The next step is to measure how often each one happens. Knowing that tool use is the most common planning failure is a claim about frequency, and frequency can only be checked by counting occurrences across many runs.

A single failed run tells you that a failure is possible. A count across runs tells you how much it matters. Because each category of failure can be studied on its own, an overall picture of agent quality comes from measuring each failure mode and then looking at them together.

The source points to external resources for this kind of measurement, including the Berkeley Function Calling Leaderboard, the AgentOps evaluation harness, and the TravelPlanner benchmark. It also mentions a simple benchmark that the author built to illustrate these failure modes, available in the book's GitHub repository. The source does not describe how these resources work in detail, so treat them as places to look rather than as descriptions of their methods.