Single-Turn and Multi-Turn Training Data

Suppose you want an AI model to use tools, such as a web search, a booking system, or a spreadsheet calculation. A natural first idea is to watch people do the job, record what they do, and use that record as training material. If a person can find a train route, the reasoning goes, then a faithful record of that person's steps should be the best teaching material available. This idea is not entirely wrong, but treating human behavior as the ideal target is the mistake this article is about.

A model's tool use ability can be improved by showing it tool use examples. Those examples are typically written with domain experts, who know which tasks need which tools. What the examples look like, and where their steps come from, shapes what the model learns.

Human annotations are not automatically ideal for AI agents. What is efficient for a human may not be efficient for a model. The efficiency of an action depends on who performs it, and a human and a model often have very different means of doing the same task. A human may prefer a visual web interface, while a model can often call an API directly. Those two preferences produce very different sequences of steps for the same goal.

Adjustment changes the form of the demonstration, not the knowledge behind it. The diagram above is a generated illustration of the idea: the human's browser routine is replaced by a single request that the agent can make and that returns usable results.

Domain experts remain valuable. The recommended approach is to have experts write tool use data, where each prompt is a task that needs tools and each response is the set of actions that completes it. Experts know what the tasks are and which tools they require. The caution applies to the form of the demonstration.

Experts can also leave steps out when they explain their work. They may forget a step, or judge it unimportant even when it changes the outcome. For this reason it is often necessary to observe how humans perform these tasks in order to ensure accuracy. Observation catches what the explanation omits.

Before collecting data, you must decide how the interaction should be structured. Whether an application needs single-turn data, multi-turn data, or both depends on how many exchanges a task requires before it is finished.

Ordinary conversation data follows a simple rhythm. The user speaks, the AI replies with one message, and the turns alternate. Tool use breaks that rhythm. The AI may need to produce several messages within a single turn, and each message may be directed to a different destination. In one example, an AI sends one message to a code interpreter, which runs the code, and another message to the user explaining what it is doing.

The comparison above shows what changes. A single-turn example pairs one user message with one AI reply. A multi-turn example can contain several AI messages inside one turn, and the destination of each message must be recorded. The extra structure is what lets training data show that one AI turn sent a message to a tool and a different message to the user.

Follow a single exchange from start to finish. The order of messages matters, because each message is sent to a specific destination. The sequence below traces a tool call. The user asks a question, the model sends code to a tool, the tool returns a result, and the model then answers the user using that result.

Notice that mess 2 and 4 come from the same AI turn, yet they travel to different places. A training pipeline that stores each turn as one undivided message would lose the destination information the model needs to learn. A plain alternating transcript cannot show which message goes where.

To handle multi-message turns, the Llama 3 authors (Dubey et al., 2024) designed a multi-message chat format. It has two main parts. Message headers specify the source and destination of each message. Special termination tokens mark where the human and AI turns begin and end. Together these let training data express which message goes where.

Curation is often pictured as adding material, but data curation works in two directions. It means creating new data to help a model learn new behaviors, and removing existing data to help a model unlearn bad behaviors. Removal matters as much as creation.

Curation follows three criteria: data quality, data coverage, and data quantity. Each application may need data with different characteristics, and different training phases may need different data mixes, but these three criteria apply throughout.

The source offers a cooking analogy. Think of model training as cooking, where the data fed into the model are the ingredients. Data quality is the quality of the ingredients, since a meal cannot be good if the ingredients are spoiled. Data coverage is the right mix of ingredients, such as not using too much or too little sugar. Data quantity is how many ingredients you should have.