Evaluating Agent Reliability

Suppose someone hands you an AI agent and asks what it is able to do. Two questions come up almost immediately. First, where does it operate: what world does it perceive and affect? Second, what can it actually do inside that world? These two questions are linked. The world an agent works in shapes the things it could possibly do, and the things it is equipped to do shape how much of that world it can reach.

This article follows that link. It then looks at how an agent works through a task one step at a time, and why long chains of steps are risky. By the end, you should be able to trace an agent's run from start to finish and explain why reliability gets harder as the number of steps grows.

The easiest way to identify an agent's environment is to ask what it is supposed to act upon. An agent built to play a game has that game as its environment. An agent that scrapes documents from the internet has the internet as its environment. A cooking robot works in a kitchen. A self-driving car works in the road system and the areas beside it. Each of these use cases names a different world, and each world contains different objects, states, and possible moves.

Once the environment is fixed, it determines which tools an agent could potentially use. A chess game is a useful case. The only possible actions for an agent inside a chess game are the legal chess moves. Nothing outside the rules of the game is available to it, so there is no point in giving it a tool for browsing the web while it plays.

A model that only produces text has a small set of possible actions: it can produce text. Giving it tools adds to that set. Many generative AI applications people use every day are agents with access to tools, although many of those tools are simple. ChatGPT is one example. It can search the web, run Python code, and generate images. Each of these capabilities is an action that is unavailable to a model with no tools at all.

Retrieval-augmented generation (RAG) systems follow the same pattern. A RAG system can be viewed as an agent whose tools are a text retriever, an image retriever, and a SQL executor. Each tool is a specific capability the agent can call on, and each one extends what the agent can do with information it finds.

A coding agent called SWE-agent, built on top of GPT-4, shows both ideas together. Its environment is a computer with a terminal and a file system. Its set of actions includes navigating a repository, searching files, viewing files, and editing lines. The environment supplies the places where those actions apply, and the actions are what let the agent change that environment.

The relationship runs in both directions, and the diagram shows both. The environment limits the tools that make sense, and the tools limit the environments the agent can actually operate in. A robot whose only action is swimming is confined to a water environment, because on land it has nothing it can do. The tools fix the reachable world just as the world fixes the usable tools.

An agent is typically given a task by a user, and it does not simply return one answer. It processes the task together with feedback from its environment, plans a sequence of actions, and decides when the task is finished. The best way to see this is to follow a real run.

Take a RAG system working with tabular sales data. It has three actions: generating a response, generating a SQL query, and executing a SQL query. Given the request to project sales revenue for a product over the next three months, the agent carries out a sequence of moves. It reasons about what information it needs, writes a query, executes that query against the data, reads what comes back, and then decides whether it has enough to answer. If the first round of results is not sufficient, it writes and runs a second round of queries.

The sequence below follows the messages between the user, the agent, the SQL tool, and the sales data. Read it top to bottom. Each arrow is a choice the agent made based on what it had just learned, and the second round of queries exists only because the first round's output was judged insufficient.

Two details in this run matter. First, the agent judges its own progress. It decides when the information is enough and when the task is complete. That judgment is part of the loop, not an afterthought, and it is what allows the agent to stop. Second, the agent returns to reasoning after every tool call. The results of each action become input for the next decision.

The sales run above had seven reasoning-and-action moments. Each one is a place where the agent can go wrong. A query can be malformed, a result can be misread, or a plan can be poor. Because the agent needs many steps, its overall accuracy depends on how the individual steps combine, not only on how good any single step is. This is called the problem of compound mistakes.

The effect is easiest to see with numbers. If a model is 95% accurate per step, then over 10 steps the overall accuracy drops to about 60%. Over 100 steps it falls to about 0.6%. The timeline below lays out those three points so you can see how quickly the trend bends downward.

Compared with non-agent uses of a model, agents typically require more powerful models. There are two reasons. The first is the compound-mistake effect just described: a long chain of steps needs each step to be highly accurate. The second is that agents with access to tools can perform more impactful tasks, but any failure could carry more severe consequences. A text answer that is wrong can be ignored. An action that is wrong may have already changed something in the environment.

Cost works in the other direction too. A task that requires many steps takes time and money to run. The source argues that this cost can still be worthwhile, because autonomous agents can save a great deal of human time. Whether an agent is worth deploying depends on weighing that saving against the expense of running many steps and the damage a failure could cause.

The environment, the tools, and the planner work as one system. The environment is set by the use case, the tools determine which actions are available inside that environment, and the planner decides which actions to take and when the task is done. Each extra step in the plan adds a chance of error, which is why accuracy falls over long tasks and why agents tend to need stronger models than simpler applications.