Evaluating Regression Models

A common first instinct when evaluating a classifier is to report one number, the fraction of predictions it gets right, and to treat a high number as proof that the model is good. The instinct feels reasonable, and it is often exactly where an evaluation goes wrong. This article follows a model through the evaluation process so you can see how a number that looks excellent can still fail at the job it was built to do, and what should replace a single accuracy score as the basis for judging a model.

Before choosing a metric, the starting question is what the application is actually for. A machine learning model is rarely the end product. Its predictions feed into a larger decision-making process, so the real interest lies in what those decisions achieve in the world. The high-level purpose of the application is called the business metric. It names the outcome an organization cares about, not the mathematical score used to check a model.

Business metrics come in many forms. Depending on the application, the goal might be avoiding traffic accidents, reducing the number of hospital admissions, attracting more visitors to a website, or getting shoppers to spend more money. Each of these goals describes a result that matters outside the model. The model is valuable only to the extent that it moves that result in the right direction.

Changing which model an application uses does not change the business metric directly. The change travels through a chain. The model produces predictions, the application turns those predictions into decisions, and those decisions produce outcomes that the business measures. Each link in that chain is a place where a model change can help, do nothing, or break the connection to the goal.

The difficulty is that the business impact of a model is often measurable only after it has been deployed in a real system. Deployment is usually the most expensive and risky point at which to find out whether the model was a good choice. The sensible move is to want an evaluation that reflects the business impact before deployment, even though the business metric itself is not yet available to read.

During early development, and whenever a model is being tuned, it is often impractical to deploy it to the real world just to see how it performs. When the possible harm is high, such as a safety failure or a serious financial loss, testing in production is not an option. The usual answer is a surrogate evaluation procedure. A surrogate evaluation uses a metric that is easier to compute than the business metric, and that metric serves as a stand-in for it.

A surrogate metric is a stand-in. It is used when the business metric cannot be checked directly during offline evaluation, either because measuring it would require deployment or because the harm of finding out by deployment is too high. The surrogate is chosen because it is feasible to compute, not merely because it is convenient.

A surrogate is only a stand-in, so it deserves care. The guidance is to find the metric that comes as close as possible to the original business goal while still being feasible to evaluate. Whenever it can be used, that closest feasible metric should be the one used for model evaluation and selection. A surrogate that is easy to compute but drifts far from the business goal makes the evaluation look reliable while measuring something the application does not need.

The outcome of such an evaluation also need not be a single number. An algorithm might bring in ten percent more customers while each one spends fifteen percent less. The evaluation should still reflect the expected business impact of choosing one model over another, which means the trade-off between those effects matters, not just one headline figure.

Binary classification, where each example is either in a positive class or a negative class, is arguably the most common and conceptually simple application of machine learning. The positive class is the one we are looking for, such as images that contain a pedestrian. Even in this simple setting, accuracy can hide important information. The counts of the four outcomes show what the single accuracy number is averaging over.

Accuracy can mislead in binary classification, especially when the positive class is rare, because a model can score highly while missing every positive case. The accuracy figure in that situation is correct as arithmetic but misleading as a measure of usefulness. The rarer the positive class, the more a model can gain in accuracy by ignoring it.