Planning for Foundation Model Agents

Picture a model working entirely on its own. A language model can produce text, and an image generator can produce pictures. Each one is good at a single kind of output. Neither model, by itself, can check the weather, look up a bank balance, or send a message. Its reach ends where its own inputs and outputs end.

An agent is different because it can be connected to things outside the model. A system does not need external tools to count as an agent, but without them its capabilities stay narrow. The question this article follows is this: once an agent can reach beyond the model, what kinds of tools does it need, and what does each kind change about the agent's behavior?

Adding tools widens what the agent can accomplish, but a larger inventory is harder to understand and harder to use well. Choosing the right inventory usually takes experimentation rather than a single correct answer. A designer is balancing reach against clarity: each added tool expands the set of possible actions, and each one also gives the agent another option to choose among.

Read-only tools let a model draw information from its data sources. Write tools let the same kind of system change those sources. The difference is easiest to see with a single tool that does both. A SQL executor can retrieve a data table, which is a read, but it can also change or delete that table, which is a write. An email API can read a message and can also send a reply. A banking API can report a current balance and can also initiate a transfer.

The comparison separates the two directions. A read flows from the data source back to the agent, so the agent gains information without changing anything. A write flows from the agent outward into the environment, so the agent's decision has consequences that persist after the call finishes. Write actions are what make automation possible. An agent could research prospective customers, draft and send emails, read the replies, follow up, record orders, and update a database, all without a person handling each step.

A workplace comparison helps. A new intern should not be given authority to delete the production database, and an unreliable AI should not be allowed to start bank transfers. The point is not that write access is forbidden. The point is that it must be matched to how dependable the system is. Humans make mistakes too, and tools are what let both humans and models accomplish far more than they could alone.

Once an agent's inventory is in view, its tools can be sorted into three families. The first is knowledge augmentation, sometimes called context construction, where tools enrich what the agent knows at the moment it is working. The second is capability extension, where tools cover tasks a model handles poorly by itself. The third family is environment interaction: tools that let the agent act upon its environment, which corresponds to the write actions described above.

These families are not mutually exclusive. A single tool can serve more than one purpose, and an agent's inventory will often mix all three. The sections that follow take the first two families, which focus on what the agent knows and what it can compute, before turning to the planner that coordinates them.

The model alone performs one kind of action. With tools, the same model can gather information from outside its own context, perform narrow computations it does poorly by itself, and change external systems. The model still does the reasoning, but the tools supply what the model cannot supply on its own.

Tools only help if the agent knows which ones to use and in what order. That is the job of planning, which sits at the heart of a foundation model agent. The model is responsible for solving a task, and a task is defined by its goal and its constraints. The goal says what must be achieved, and the constraints limit how it may be achieved.

Complex tasks cannot be handled in a single leap, so the agent produces a plan. A plan is a roadmap that lays out the steps needed to accomplish the task. Producing a good plan usually means understanding the task, considering several ways to achieve it, and choosing the most promising one. For the purposes of tool use, the key point is that the planner is the component that decides which tools to call and in what order.

A model that handles only one kind of input or output is called unimodal. Tools can change that. A text-only model can gain image generation by calling a text-to-image model as a tool, so the combined system produces both text and images. The model itself never learns to draw. It learns to recognize when a drawing is needed and to hand the request to a tool that can draw.

Read the diagram as a control path. The planner reads the incoming request and decides whether text generation, image generation, or both are needed. Either branch can run alone, and when the request calls for both, their outputs are combined into one reply. The same routing pattern explains how ChatGPT produces text and images, since it relies on DALL-E as its image generator.

The idea runs in the other direction too. A model that accepts only text can call an image captioning tool to interpret pictures, a transcription tool to handle audio, or an OCR tool to read text out of PDFs. Other tool chains extend the same logic: a code interpreter can produce charts and graphs, a LaTeX compiler can render math equations, and a browser can render web pages from HTML.