Data Acquisition and Annotation

When a finetuning project stalls because there is not enough labeled data, the natural instinct is to ask for more. Collecting more examples feels like the safest move, because a bigger pile of training material seems certain to help. That instinct is often right in direction, but it hides a question that matters a great deal in practice: how much more data is actually needed, and at what cost?

This article looks at two things. The first is how you can answer the question of how much more data you need before you spend money on collection. The second is how data diversity and the annotation budget limit how much of your data a project can realistically use. Together these explain why the decision to collect more data should be measured rather than assumed.

The practical way to test the equal-gains belief is to measure it on your own task, and you do not need to collect anything new to begin. Take the dataset you already have, train on a fraction of it, and record how well the model performs. Repeat this at several fractions. A common design uses quarters and halves of the data alongside the full set, which gives you three or four measurements that you can compare directly.

Once you have the measurements, plot performance gain against dataset size. The shape of that plot tells you what more data is likely to do. A steep, rising relationship suggests that doubling the data would bring a meaningful improvement. A flat stretch suggests that doubling would bring only a small one. The plot is therefore a forecast tool: it lets you reason about a larger dataset without having one yet.

The curve you produce usually bends. Early points often climb quickly, and later points climb more slowly. This pattern is called diminishing returns. The same number of added examples typically buys a smaller improvement once the dataset is already sizable. When the curve is still steep at the right edge, you have evidence that more data will pay off. When it has flattened, the evidence points the other way, and spending heavily on collection may not be justified by the task alone.

Counting examples is only part of the picture. The variety of what the examples cover also affects performance. In a study of instruction finetuning, performance rose sharply as the number of finetuning tasks grew from 9 to 282. Beyond about 282 tasks, gains began to level off, though small positive improvements continued up to 1,836 tasks. A model benefits a great deal from exposure to many different kinds of work, not simply from a larger pile of similar work.

Diversity can be considered along three dimensions. The first is the types of task, such as summarization or question answering. The second is the topics covered, such as fashion, finance, or technology. The third is the output formats expected, such as structured JSON or yes-or-no answers. A dataset that is large but narrow along all three dimensions may plateau early, while a smaller dataset spread across them may teach the model more.

Even when the performance curve argues for more data, the amount you can use is limited by money. Annotation has a per-example cost, so a fixed budget sets a maximum number of labeled examples. The calculation is direct: divide the budget by the cost of labeling one example. The result is an upper bound on what you can annotate, not a target you must reach.

Money also has to be split between data and compute. Spending more on annotation leaves less for training runs, and the reverse is also true. A team that pours its entire budget into labeling may find it cannot afford the experiments needed to use that data well. Planning therefore means deciding both how many examples to buy and how much computation to reserve for finetuning them.

Sometimes the bottleneck is high-quality, task-specific data itself. One strategy is to finetune first on data that is lower in quality or less relevant, and only then finetune on the scarce target data. This can reduce how much high-quality data is needed. The source gives three patterns that illustrate the idea.