Training Data Requirements for Model Performance

When you first learn that large language models are more capable than small ones, a natural guess follows: a more capable model should be able to learn from fewer examples. If the model is smarter, surely it needs less help from labeled data. The belief feels reasonable, and it quietly shapes how many people estimate the cost of building an AI application. It is also wrong in the way that matters most for budgeting.

In this article you will trace what happens to the data requirement as a model grows, then trace how the price of labeling is built up from its parts, and finally check your own budget arithmetic. Keep one question in mind throughout: what is the quantity you are actually measuring?

The key to the puzzle is what the comparison is measuring. One way to test a large model is to give it the same data a small model received and ask whether it does as well. That question is not the one builders usually care about. A builder wants the strongest performance the application can reach, so the relevant question is how much data the large model needs to reach its own best performance.

Seen this way, the answer changes. A large model has more capacity to absorb patterns, and feeding it only the small dataset leaves much of that capacity unused. Scaling up the model is therefore usually paired with scaling up the training data, so that the bigger model has enough examples to make use of its extra size.

Labeling data is a service with a price, and that price is not fixed across projects. The actual cost of labeling depends on three main factors. The first is the complexity of the task: how difficult it is for a labeler to make each judgment. The second is the scale of the dataset, since larger datasets typically come with a lower cost per sample. The third is the labeling service provider you choose, because providers set their own prices.

Notice that scale behaves differently from the other two factors. Complexity and provider choice change the price you pay for each sample. Scale tends to push the per-sample price down, but it also means you are buying more samples in total. Per-sample price and total budget are separate quantities, and a good estimate tracks both.

Amazon SageMaker Ground Truth offers a concrete illustration of volume pricing. As of September 2024, it charged 8 cents per image for labeling fewer than 50,000 images, and only 2 cents per image for labeling more than 1 million images. The price per image drops by a factor of four as the project grows from the smaller tier to the larger one.

Quoted per-sample prices must be multiplied by the number of samples, and the price tier a project falls into matters. The two quantities are linked by two simple relationships. The total cost is the price per sample multiplied by the number of samples, and the per-sample cost is the total divided by the number of samples.

Here C_total is the total labeling cost, p is the price per sample, and N is the number of samples. Read the second form as a check: dividing a total by the number of samples recovers the per-sample price you were quoted.

The errors below come from treating the data requirement and the labeling price as simple, fixed numbers. Each one is easy to make when a plan is first sketched.