Cost-Benefit Analysis in Model Selection
An ordering is a relative verdict. Two models can be ordered correctly while both fall short of what the task requires, so the ranking by itself says nothing about adequacy.
The Habit of Trusting the Higher Score
A common habit in model selection goes like this. You run two candidate models on the same test set, one of them scores higher, and you conclude that the higher-scoring model is the one to use. The reasoning feels airtight. The winner is better, so the winner should be deployed.
This article examines the gap hidden inside that reasoning. A ranking tells you which option comes out ahead. It does not tell you whether the option that came out ahead is actually good enough for the job you need it to do. Keeping those two questions separate is the core skill of cost-benefit thinking about models.
Two Models, One Ordering
Suppose a comparison says model B is better than model A. That verdict is relative: it describes the order of the two, and nothing else. The limitation is built into that kind of result, because several different realities could produce that exact same ranking.
In one reality, model B is solid while model A is poor. In a second, both models are poor, with B only slightly less poor. In a third, both are strong, with B edging ahead. The ordering is identical in all three cases, yet the practical meaning is completely different. A comparison that always returns an ordering cannot distinguish between these worlds on its own.
An ordering is a relative verdict. Two models can be ordered correctly while both fall short of what the task requires, so the ranking by itself says nothing about adequacy.
Comparative Versus Absolute Answers
Comparative evaluation answers the question of which model is better. Absolute performance answers the question of how well a model performs on its own terms and whether it is adequate for the use case.
The two questions look similar but call for different evidence. Asking which model is better requires only that the models be run on the same test and their results be lined up against each other. Asking whether a model is good enough requires a measurement of that model by itself, judged against what the task needs. A preference result can be perfectly accurate and still leave the second question unanswered.
Treating a comparison result as a verdict on adequacy is the central misconception here. A comparison can tell you which option wins, but only absolute measurement tells you how well the winner performs.
Tracing a Customer Support Swap
Consider a concrete setting. Model A handles customer support tickets and resolves 70% of them. That number is an absolute measurement, because it describes model A on its own terms. Now bring in model B, which wins against model A in 51% of head-to-head comparisons.
Reading the Support Ticket Numbers
Model A resolves 70% of customer support tickets. Model B wins against model A in 51% of head-to-head comparisons. How many tickets would model B resolve if deployed?
- 1Identify the absolute figure
The 70% resolution rate describes model A on its own terms. It is an absolute measurement of model A.
- 2Identify the comparative figure
The 51% figure records how often model B is preferred over model A in head-to-head comparisons. It is a relative measurement.
- 3Check whether the figures measure the same thing
They do not. The 51% tells you how often B is preferred. It does not tell you how many tickets B would resolve.
- 4Look for the missing link
To get from the preference figure to a resolution figure for B, you would need some additional evaluation that links the two. Nothing in the two numbers supplies that link.
The problem cannot be solved from the given numbers. Model B's resolution count is unknown, so the two figures cannot be combined into one.
Why a Swap Needs an Absolute Baseline
Choosing between models is rarely decided by quality alone. Human preference matters, but so do other considerations, and cost is one of the most important. A cost-benefit analysis weighs what a swap would gain against what it would add in expense.
The difficulty is that the gain side of that comparison requires knowing how much performance will actually improve. Comparative evaluation can say that B is preferred over A, but it does not supply the size of the performance boost that would result from switching. Without that magnitude, the expense side has nothing to be weighed against.
The source makes this concrete with a pointed case. If model B costs twice as much as model A, then knowing B is preferred is not sufficient. Whether the extra expense is justified depends on how large the performance improvement is in absolute terms. Without that magnitude, the cost-benefit judgment is guesswork dressed up as analysis.
Testing the Swap With Hypothetical Numbers
The following numbers are invented purely for illustration and are not drawn from any real system. Suppose a team learns that model B costs twice as much as model A per ticket, and that B is preferred in most head-to-head comparisons. The team is tempted to switch.
A Swap Judged With and Without the Gain Figure
Model B costs twice as much as model A. Should the team switch? Work through the decision twice, once with only the preference result and once with an absolute performance gain added.
- 1Case one: only the preference result
The team knows B is preferred over A. It knows B costs twice as much. It does not know how many more tickets B would resolve, so it cannot weigh the extra cost against any benefit.
- 2Case two: an absolute gain is measured
Imagine a separate evaluation shows that B resolves a small number of additional tickets per month. The team can now set that gain against the doubled cost.
- 3Judge the trade
With the gain known, the team can ask whether the extra resolved tickets are worth the extra expense for its own use case. The question has a real answer only in case two.
The preference result alone supports no confident decision. Only the absolute gain makes the cost-benefit judgment possible.
Mistakes That Hide the Missing Baseline
Common mistake Treating the winner as good enough to deploy
What it looks likeModel B beats model A on the shared test set, so the team ships model B.
Why it failsA ranking is relative. It shows which option comes out ahead, not whether that option meets the job's requirements.
Do this insteadPair the ranking with an absolute measurement of the winner and check that measurement against what the task needs.
Common mistake Converting a preference rate into an outcome count
What it looks likeSince B wins 51% of comparisons, the team assumes B resolves 51% of tickets.
Why it failsThe 51% measures how often B is preferred. It does not measure how many tickets B would resolve.
Do this insteadRun an evaluation that measures the outcome directly for the model being considered.
Common mistake Ignoring cost when the performance gain is unknown
What it looks likeModel B is preferred, so the team accepts its higher price without estimating the benefit.
Why it failsA cost-benefit judgment requires the size of the gain. Without it, the expense has nothing to be weighed against.
Common mistake Assuming identical rankings mean identical practical quality
Checking a Swap Decision
What do you think happens?
Model A resolves 70% of support tickets. Model B wins 51% of head-to-head comparisons against model A. Before reading on, predict: does this mean model B resolves 51% of tickets?
Reveal answer
No. The 51% measures how often B is preferred, not how many tickets B would resolve.
Converting the preference figure into a resolution figure would require additional evaluation that links the two measurements.
A colleague proposes replacing model A with model B because B wins most head-to-head comparisons. Explain what information you would need before you could make a confident cost-benefit decision, and say which of the two questions, which is better or is it good enough, that information answers.
Hints
- Separate the preference result from the absolute performance of each model.
- Think about what the gain side of the cost-benefit comparison requires.
- Consider what evidence would measure B's performance on its own terms.
Key Takeaways
- A comparative ranking identifies which model is better but does not establish that either model is good enough for a given use case.
- Several different underlying realities can produce the same ranking, so the ordering alone cannot separate strong models from weak ones.
- Comparative evaluation answers which model is better, while absolute performance answers how well a model performs on its own terms and whether it is adequate.
- A win rate does not convert into a number of successful outcomes, and cost-benefit judgments about a swap require the size of the absolute performance gain.