Choosing Evaluation Metrics for Multiclass and Regression Tasks

When a learner first trains a classifier, the natural habit is to look at one number, usually accuracy, and pick the model with the highest value. That habit feels rigorous, and in many small exercises it works. This lesson examines the belief behind that habit: that the metric you compute during development is the goal itself. That belief is usually wrong, and seeing why is the first step toward choosing a metric well.

The misconception behind the accuracy habit is this: if model A has a higher evaluation score than model B, then model A is the better choice for the application. Under this belief, the evaluation score is the target, and the job of machine learning is to push that number as high as possible. The correction is that people are interested not just in making accurate predictions, but in using those predictions as part of a larger decision-making process. The prediction is an input to a decision, and the decision is what has consequences in the world.

Before picking a machine learning metric, think about the high-level goal of the application. This goal is called the business metric. Examples include avoiding traffic accidents, decreasing the number of hospital admissions, getting more users for a website, and having users spend more money in a shop. Each is stated in the language of the organization or the people the system serves, not in the language of a model. A prediction score, by contrast, is a number computed from the model's outputs on a test set. The business metric is what you want to improve; the prediction score is a measurement you can compute during development.

The consequences of choosing a particular algorithm for a machine learning application are called the business impact. The distinction matters. The business metric is what you want to improve. The business impact is what actually happens to that goal when you deploy one algorithm instead of another. Assessing the business impact of a particular model might require putting it into a real-life system, which is often difficult to do.

Business impact can also be more than a single number. Consider a model whose consequence is that you gain 10 percent more customers, but each customer spends 15 percent less. The evaluation should still capture the expected business impact of choosing one model over another, which means combining these effects rather than looking at either one alone.

Why not simply measure business impact directly? In early development, and when tuning parameters, it is often infeasible to put models into production just for testing, because the business or personal risks can be high. Consider evaluating the pedestrian avoidance of a self-driving car by letting it drive around before verifying it. If the model is bad, pedestrians are the ones who suffer.

The remedy is a surrogate evaluation procedure: an evaluation metric that is easier to compute than the business metric. For the pedestrian case, one surrogate is to test whether the model can classify images of pedestrians versus non-pedestrians, and then measure accuracy on that test. The surrogate is a stand-in that lets you learn something before the expensive or dangerous real-world test.

A surrogate is only a surrogate, so it pays off to find the metric closest to the original business goal that is still feasible to evaluate. That closest metric should be used whenever possible for model evaluation and selection. A surrogate that is easy to compute but far from the goal can lead you to choose the wrong model with great confidence.

Binary classification is arguably the most common and conceptually simple application of machine learning in practice, yet even this task has caveats. For binary classification we speak of a positive class and a negative class, where the positive class is the one we are looking for. Measuring accuracy can be misleading, and the choice of metric should reflect the positive class and the real goal rather than a single score.