Binary Classification Metrics

When you first train a classifier, the natural habit is to look at one number, usually accuracy, and pick the model with the highest value. That habit feels rigorous, and in many small exercises it works. This article examines the belief behind the habit: that the metric you compute during development is the goal itself. The source material argues that this belief is usually wrong, and it explains what should replace it.

The misconception is this: if model A has a higher evaluation score than model B, then model A is the better choice for the application. Under this belief, the evaluation score is the target, and the job of machine learning is simply to push that number as high as possible. The source corrects this directly. In practice, people care about more than accurate predictions. They use those predictions as part of a larger decision-making process. The prediction is an input to a decision, and the decision is what has consequences in the world.

Binary classification is the task of sorting every case into one of two classes. The source calls it arguably the most common and conceptually simple application of machine learning in practice. We speak of a positive class and a negative class, where the positive class is the one we are looking for. Even this simple task has a caveat: measuring accuracy can be misleading when the classes are unequal in size.

Trace the problem with a concrete dataset. Suppose there are 1000 images, and only 10 of them are positives, the class we are looking for. The other 990 are negatives. Now imagine a model that always predicts negative and never looks at the image at all. Follow what happens to its results.

The source gives examples of business metrics: avoiding traffic accidents, decreasing the number of hospital admissions, getting more users for a website, and having users spend more money in a shop. Each one is stated in the language of the organization or the people the system serves, not in the language of a model. None of them is a number a classifier outputs.

A prediction score is what the model produces during development, such as an accuracy value on a test set. It is a proxy you compute to help decide between models. The distinction matters: the business metric is what you want to improve, and the prediction score is a measurement you use to guess how well a model will help improve it. The two can point in different directions, as the always-negative model showed. A high prediction score can coexist with no progress on the business metric at all.

The source notes that business impact may only be fully assessed by putting a model into production. Assessing it for a particular model can therefore require placing that model into a real-life system, which is often difficult to do. The consequences of a choice are real outcomes, and they show up only when the choice meets the world.

The worked example shows why the source concludes that evaluation should still capture expected business impact. The result of evaluating a model is not always a single number that maps directly onto the goal. Two effects can move in opposite directions, and only the combined consequence tells you which choice is better.

Why not simply measure business impact directly during development? The source explains that in early development, and when tuning parameters, it is often infeasible to put models into production just for testing, because the business or personal risks can be high. The source gives a vivid case: evaluating the pedestrian avoidance of a self-driving car by letting it drive around without first verifying it. If the model is bad, pedestrians are the ones who suffer.

Follow the pedestrian example through the pipeline. You cannot yet measure how well the car avoids pedestrians on real roads. So you test whether the model can classify images of pedestrians versus non-pedestrians, and you measure accuracy on that test. The result lets you rank candidate models today, without risking anyone. The real-world test remains the eventual judge of business impact.

The source states that the closest feasible metric to the business goal should be preferred, and that this closest metric should be used whenever possible for model evaluation and selection.

Read the chain from goal to measurement: the business metric names the goal, the business impact is what happens to that goal when you deploy a chosen algorithm, and the surrogate evaluation procedure is the feasible stand-in you use while the real test is too risky or too costly to run. Each link is a place where the number on your screen can drift from what matters in the world.