Choosing Models for Application Tasks

When you first learn that large language models are more capable than small ones, a natural guess follows: a more capable model should be able to learn from fewer examples. If the model is smarter, surely it needs less help from labeled data. This belief feels reasonable, and it shapes how many people estimate the cost of building an AI application. It is also wrong in the way that matters most for budgeting. This article explains why, and then shows how to turn a labeling quote into a realistic budget.

The key to the puzzle is what the comparison is measuring. One way to test a large model is to give it the same data a small model received and ask whether it does as well. That question is not the one builders usually care about. A builder wants the strongest performance the application can reach, so the relevant question is how much data the large model needs to reach its own best performance.

Seen this way, the answer changes. A large model has more capacity to absorb patterns, and feeding it only the small dataset leaves much of that capacity unused. Scaling up the model is therefore usually paired with scaling up the training data, so that the bigger model has enough examples to make use of its extra size.

Labeling data is a service with a price, and that price is not fixed across projects. The actual cost of labeling depends on three main factors. The first is the complexity of the task: how difficult it is for a labeler to make each judgment. The second is the scale of the dataset. The third is the labeling service provider you choose, since providers set their own prices.

Scale behaves differently from the other two factors. Complexity and provider choice change the price you pay for each sample. Scale tends to push the per-sample price down, because larger datasets typically come with a lower cost per sample. But scale also means you are buying more samples in total. The per-sample price and the total budget are separate quantities, and a good estimate tracks both.

Amazon SageMaker Ground Truth offers a concrete illustration of volume pricing. As of September 2024, it charged 8 cents per image for labeling fewer than 50,000 images, and only 2 cents per image for labeling more than 1 million images. Prices change over time, so these figures are a snapshot of one provider at one date rather than a permanent rate.

Quoted per-sample prices must be multiplied by the number of samples, and the price tier a project falls into matters. The two quantities are linked by two simple relationships. The total cost is the per-sample price multiplied by the number of samples. Going the other way, the per-sample cost is the total cost divided by the number of samples.

The following errors appear when teams plan labeled data. Each one comes from treating the per-sample price and the total budget as if they were the same quantity, or from assuming the quoted price never changes.