Function Calling in Language Models

Picture a model working entirely on its own. A language model can produce text, and an image generator can produce pictures. Each one is good at a single kind of output. Neither model, by itself, can check the weather, look up a bank balance, or send a message. Its reach ends where its own inputs and outputs end.

An agent is different because it can be connected to things outside the model. A system does not need external tools to count as an agent, but without them its capabilities stay narrow. The central question for this article is this: once an agent can reach beyond the model, what kinds of tools does it need, and what does each kind change about the agent's behavior?

Tools do two jobs. Some let an agent take in information about its surroundings, and others let it change something in those surroundings. Both jobs matter. An agent that can only observe is stuck watching, and an agent that can only act has no picture of what it is acting on.

Read-only tools let a model draw information from its data sources. Write tools let the same kind of system change those sources. The difference is easiest to see with one tool that does both. A SQL executor can retrieve a data table, which is a read, but it can also change or delete that table, which is a write. An email API can read a message and can also send a reply. A banking API can report a current balance and can also initiate a transfer.

The two directions behave differently. A read flows from the data source back to the agent, so the agent gains information without changing anything. A write flows from the agent outward into the environment, so the agent's decision now has consequences that persist after the call finishes. Write actions are what make automation possible. An agent could research prospective customers, draft and send emails, read the replies, follow up, record orders, and update a database, all without a person handling each step.

A helpful comparison is the workplace. A new intern should not be given authority to delete the production database, and an unreliable AI should not be allowed to start bank transfers. Humans make mistakes too. Tools are what let both humans and models accomplish far more than they could alone.

Once an agent's inventory is in view, its tools can be sorted into three families. The families are not mutually exclusive. A single tool can serve more than one purpose, and an agent's inventory will often mix all three. The map below shows how the families relate to the agent they extend.

A tool call is not a single event. It is an exchange among three parties: the model, a runtime that handles the call, and the external tool. The sequence below is a generic illustration of this round trip, showing one request from start to finish. Read it as a pattern rather than as a fixed protocol.

Tools only help if the agent knows which ones to use and in what order. That job belongs to planning. The model is responsible for solving a task, and a task is defined by its goal and its constraints. The goal says what must be achieved, and the constraints limit how it may be achieved. Complex tasks cannot be handled in a single leap, so the agent produces a plan: a roadmap that lays out the steps needed to accomplish the task.

The planner is the component that decides which tools to call and in what order. In a multimodal setting, the planner reads the incoming request and decides whether text generation, image generation, or both are needed. Either branch can run alone, and when the request calls for both, their outputs are combined into one reply. The same routing pattern explains how ChatGPT produces text and images, since it relies on DALL-E as its image generator.

The idea runs in the other direction too. A model that accepts only text can call an image captioning tool to interpret pictures, a transcription tool to handle audio, or an OCR tool to read text out of PDFs. Tool use can also outperform the alternatives: the Chameleon system pairs a GPT-4-powered agent with 13 tools and outperforms GPT-4 alone on several benchmarks. On ScienceQA, Chameleon improves on the best published few-shot result by 11.37%. On TabMWP, a benchmark of tabular math word problems, it improves accuracy by 17%.