Choosing Evaluation Metrics for Multiclass Classification

A common first instinct when evaluating a classifier is to report one number, the fraction of predictions it gets right, and treat a high number as proof that the model is good. The instinct feels reasonable, and it is often exactly where an evaluation goes wrong. This article examines the belief that a single accuracy score is always the right way to judge a model, and what should replace it. The source material develops these ideas using binary classification, where each example is either in a positive class or a negative class. The principles it describes are the foundation for choosing metrics in any classification setting.

Before choosing a metric, the starting question is what the application is actually for. A machine learning model is rarely the end product. Its predictions feed into a larger decision-making process, so the real interest lies in what those decisions achieve in the world. The high-level purpose of the application is called the business metric. It names the outcome an organization cares about, not the mathematical score used to check a model.

Business metrics come in many forms. Depending on the application, the goal might be avoiding traffic accidents, reducing the number of hospital admissions, attracting more visitors to a website, or getting shoppers to spend more money. Each of these goals describes a result that matters outside the model. The model is valuable only to the extent that it moves that result in the right direction.

The consequences of picking one particular algorithm for an application are called the business impact. Choosing a model or setting its parameters should favor the option that has the most positive effect on the business metric. The difficulty is that the business impact of a model is often measurable only after it has been deployed in a real system. That is usually the most expensive and risky point at which to discover whether the choice was a good one.

During early development, and whenever a model is being tuned, it is often impractical to deploy it to the real world just to see how it performs. When the possible harm is high, such as a safety failure or a serious financial loss, testing in production is simply not an option. The usual answer is a surrogate evaluation procedure: an evaluation that uses a metric which is easier to compute than the business metric, and which serves as a stand-in for it.

Binary classification is arguably the most common and conceptually simple application of machine learning, and it is where accuracy's weakness shows up most clearly. Accuracy counts every correct prediction equally, whether it was a rare positive case or one of the many easy negatives. When the positive class is rare, the negatives dominate the count, and a model can score highly while missing every positive case. The accuracy figure is correct as arithmetic but can be misleading as a measure of usefulness.

A surrogate is only a stand-in, so it deserves care. The guidance is to find the metric that comes as close as possible to the original business goal while still being feasible to evaluate. Whenever it can be used, that closest feasible metric should be the one used for model evaluation and selection. The flow below shows how the business goal narrows down to the metric that drives the model choice.

The outcome of such an evaluation also need not be a single number. An algorithm might bring in ten percent more customers while each one spends fifteen percent less. The evaluation should still reflect the expected business impact of choosing one model over another, which means the metric must account for both effects rather than collapsing them into one figure that hides the trade-off.

The evaluation should reflect the consequences that matter to the application. Consider a model that brings in ten percent more customers while each one spends fifteen percent less. Whether that model is better depends on how the business weighs the two effects together. A single headline figure may look like a verdict while quietly ignoring one side of the trade-off, which is why the source treats a single number as possibly insufficient.