Planning in Agent Systems

Suppose you are handed an AI agent and asked what it is able to do. Two questions come up almost immediately. First, where does it operate: what world does it perceive and affect? Second, what can it actually do inside that world? These questions are linked. The world an agent works in shapes the things it could possibly do, and the things it is equipped to do shape how much of that world it can reach.

This article follows that link. It then looks at how an agent works through a task one step at a time, and why long chains of steps are risky. Along the way you will trace a concrete run, see how each step feeds the next, and see why a small error rate per step can add up to a large chance of failure.

The easiest way to find an agent's environment is to ask what it is supposed to act upon. An agent built to play a game has that game as its environment. An agent that scrapes documents from the internet has the internet as its environment. A cooking robot works in a kitchen. A self-driving car works in the road system and the areas beside it.

Once the environment is fixed, it determines which tools an agent could potentially use. A chess game is a useful case. The only possible actions for an agent inside it are the legal chess moves. Nothing outside the rules of the game is available to it.

The relationship between environment and tools runs in both directions. Start with the environment, which determines the tools that could be used. Now read the relationship the other way. The tool inventory an agent carries restricts which environments it can actually work in. If a robot's only action is swimming, it is confined to a water environment, because on land it has nothing it can do. The tools fix the reachable world just as the world fixes the usable tools.

A model that only produces text has a small set of possible actions. Giving it tools adds to that set. ChatGPT is one example: it can search the web, run Python code, and generate images. Each of these capabilities is an action that is unavailable to a model with no tools at all. Many of these tools are simple, but each one still widens what the agent can do.

Retrieval-augmented generation (RAG) systems follow the same pattern. A RAG system can be viewed as an agent whose tools are a text retriever, an image retriever, and a SQL executor. Each tool is a specific capability the agent can call on.

A coding agent called SWE-agent, built on top of GPT-4, shows both ideas together. Its environment is a computer with a terminal and a file system. Its set of actions includes navigating a repository, searching files, viewing files, and editing lines. The environment supplies the places where those actions apply, and the actions are what let the agent change that environment.

An agent is typically given a task by a user. It does not simply return one answer. It processes the task together with feedback from its environment, plans a sequence of actions, and decides when the task is finished. To see this in action, take a RAG system with tabular sales data. It has three actions: generating a response, generating a SQL query, and executing a SQL query.

Given the request to project sales revenue for a product over the next three months, the agent carries out a sequence of reasoning-and-action moments. The diagram below follows that run. Each step is a point where the agent settles on its next move, and each arrow is a choice it made based on what it had just learned. The agent returns to reasoning after every tool call, and the second round of queries exists only because the first round's output was judged insufficient.

Read the diagram from top to bottom. The user supplies the task. The planner decides on an action, the SQL executor carries it out, and the result returns to the planner. The planner then makes a decision that the source treats as part of the loop: whether the information is enough to finish. Because it was not enough in this run, the planner sent a second round of work to the tool before answering. The loop only ends when the planner judges the task complete.

Two details in this run are worth noticing. The agent judges its own progress: it decides when the information is enough and when the task is complete. The judgment is part of the loop, not an afterthought. The source describes this as the AI component deciding whether the task has been accomplished, which is what allows the agent to stop.

The sales run had seven reasoning-and-action moments. Each one is a place where the agent can go wrong: a query can be malformed, a result can be misread, or a plan can be poor. Because the agent needs many steps, its overall accuracy depends on how the individual steps combine, not only on how good any single step is. The source calls this the problem of compound mistakes.

The source gives a concrete case. If a model is 95% accurate per step, then over 10 steps the overall accuracy drops to about 60%, and over 100 steps it falls to about 0.6%. The pattern matches multiplying the 95% figure by itself once for each step. A small per-step error rate, repeated over many steps, adds up to a large chance that something has gone wrong somewhere along the way.

Compared with non-agent uses of a model, agents typically require more powerful models, and the source gives two reasons. The first is the compound-mistake effect just described: more steps means more chances for error. The second is that agents with access to tools can perform more impactful tasks, but any failure could carry more severe consequences. A text answer that is wrong can be ignored; an action that is wrong may have already changed something in the environment.

Cost works in the other direction too. A task that requires many steps takes time and money to run. The source argues that this cost can still be worthwhile, because autonomous agents can save a great deal of human time. Whether an agent is worth deploying depends on weighing that saving against the expense of running many steps and the damage a failure could cause.