Curating Data for Model Training

Imagine you want a model to find the cheapest train ticket between two cities. Before any training begins, someone has to decide what the model should learn to do. A person would typically open a browser, type the route into a search bar, and click through the results one by one. A model could instead be trained to send a single request to a search interface that returns the results directly. Both approaches reach the same goal, so the training data you collect determines which path the model learns.

This article is about that choice. A model's tool use ability can be improved by showing it examples of tool use, and the way those examples are gathered and cleaned shapes what the model learns. The sections below follow one idea from start to finish: training data is not automatically good because it came from a person doing the job, and it is not finished until it has been checked against three criteria.

A natural first idea for teaching a model to use tools is to watch people do the job and record what they do. If a person can search the web, book a room, or run a spreadsheet calculation, a faithful record of that person's steps seems like the best possible training material. The appeal is easy to understand: the human already knows how to finish the task, so copying the human looks like the shortest route to a capable model.

The source does not say this belief is entirely wrong. Its warning is narrower: treating human behavior as the ideal target is the mistake to avoid. Human steps are a useful reference point, but they are not the same thing as the steps a model should take.

The source makes a direct claim: human annotations might not be ideal for AI agents. The reason is that the efficiency of an action depends on who performs it. A human may prefer a visual web interface, because that is how a person sees and interacts with a page. A model can often call an API directly, which means it does not need to click through a page at all. Those two preferences produce very different sequences of steps for the same goal.

This means a record of human steps may describe a route the model should never take. The ticket example makes this concrete. The human routine of opening a browser and clicking result by result is efficient for a person working with a screen. For a model that can send a request to a search interface, the same routine adds steps that serve no purpose for it.

This does not mean human expertise is useless. The source recommends using domain experts to write tool use data. In this arrangement, each prompt is a task that needs tools, and each response is the set of actions that completes it. Experts know what the tasks are and which tools they require. The caution is about the form of the demonstration, not the value of the knowledge behind it.

Experts can also leave steps out when they are asked to explain their work. They may forget a step, or judge it unimportant even when it matters for the outcome. For this reason the source says it is often necessary to observe how humans perform these tasks in order to ensure accuracy. Observation catches what the explanation omits, while expertise supplies the tasks and the tools that each task calls for.

A raw demonstration usually contains more than the agent needs. It may include hesitations, detours, and steps that exist only because a person was reading a screen. Before the demonstration is used for training, it should be checked and cleaned so that what remains is the sequence the agent should actually learn. The diagram below contrasts the two versions of the same ticket search.

In the cleaned version, the browsing steps are gone because the agent reaches the same results through a direct request. The goal is unchanged. Only the path the agent is trained to follow has been replaced with one suited to the performer. This is the kind of adjustment the source has in mind when it says human demonstrations may need to be checked before they are useful for an AI agent.

Before collecting any data, you must decide how the interaction should be structured. The source asks you to consider whether your application needs single-turn data, multi-turn data, or both. The distinction depends on how many exchanges a task requires before it is finished. A task that is completed by one reply is single-turn. A task that takes several exchanges, or several messages within one reply, is multi-turn.

Ordinary conversation data assumes a simple rhythm. The user speaks, the AI replies with one message, and the turns alternate. Tool use breaks that rhythm. The AI may need to produce several messages within a single turn, and each message may be directed to a different destination. The source's example is an AI that sends one message to a code interpreter, which runs the code, and another message to the user to explain what it is doing.

Follow a single exchange in order. The user sends a request. The AI model sends a message to the code interpreter, which runs the code and returns a result. The AI then sends a second message to the user to explain what happened. Each message feeds the next, and the destination of every message is part of what the model must learn.

Notice that the AI model appears twice in the middle of the sequence. Its message to the interpreter and its message to the user come from the same AI turn. A dataset that records only the user and the AI as two parties would hide the fact that the AI's output had two different destinations.

To handle this structure, the Llama 3 authors (Dubey et al., 2024) designed a multi-message chat format. It has two main parts. Message headers specify the source and destination of each message. Special termination tokens mark where the human and AI turns begin and end. Together these let training data express which message goes where, which a plain alternating transcript cannot show.

Curation is often pictured as adding material, but the source defines it in two directions. Data curation means creating new data to help a model learn new behaviors, and removing existing data to help a model unlearn bad behaviors. Removal matters as much as creation.

The source gives a concrete case. Imagine a chatbot that users complain is arrogant. When asked to verify whether a statement is factually correct, it confirms the fact and then produces an unsolicited rewrite of the statement. Investigating the training data reveals several annotations that include unsolicited suggestions. The fix has two parts: request removal of those examples from the training data, and request new examples that demonstrate fact-checking without unsolicited rewriting.

At a high level, data curation follows three criteria: data quality, data coverage, and data quantity. Each application may need data with different characteristics, and different training phases may need different data mixes, but these three criteria apply throughout. The diagram shows how the three criteria all judge the same training decision.

The three criteria are separate questions, so a dataset can succeed on one and fail on another. A large collection of spoiled examples has quantity but not quality. A small set that contains only one kind of interaction may have quality but poor coverage. A good training decision has to hold up on all three at once.