Limitations of Accuracy Metric
Accuracy is the first number most people look at, and in many problems it is a reasonable one.
A Score That Looks Too Good
Suppose a colleague tells you their classifier is 90 percent accurate. Your first reaction is probably to congratulate them. Accuracy is the first number most people look at, and in many problems it is a reasonable one. In some problems, however, the number can look excellent while the model has done nothing that deserves credit. This article is about why that reaction is often premature.
The misconception this article addresses is that a model is good whenever its accuracy is high. The belief feels natural. A 99 percent accurate model sounds nearly perfect, and few people stop to ask what the model was being accurate about. The problems where a misleading accuracy appears share one feature: one class is far more common than the other.
The habit you will build here is simple. Before you believe a score, check what a trivial strategy would have scored on the same data. If a strategy that learns nothing already reaches a similar number, the score is not telling you much about the model.
What Makes a Dataset Imbalanced
A dataset is imbalanced when one class is much more frequent than the other. The more frequent class is often called the common or majority class, and the less frequent class is called the rare or minority class.
The difference between a balanced and an imbalanced dataset is a difference in proportions. In a balanced dataset the classes occur with equal or similar frequency. In an imbalanced dataset, one class may occupy most of the rows while the other appears only rarely. The diagram below contrasts the two situations using the click example that runs through this article.
Imbalance is not a rare edge case. Imbalanced data is the norm, and it is rare for the events of interest to occur with equal or even similar frequency. Many problems are built around events that are unusual by nature, so the rare class is often the one the user cares about most.
Click-through prediction is a typical case. Most items shown to users are not clicked, so the no-click label dominates the training data. The click label, which is the event the system is trying to anticipate, is the minority. The imbalance is not a flaw in the data collection; it is how the underlying events are distributed.
The Constant Prediction Trap
Return to the click example and consider a classifier that does nothing except output no click for every item. Because 99 percent of the samples are no click, this classifier is right 99 percent of the time. It has learned nothing about users or items. Its accuracy is high only because of how the labels are distributed.
What do you think happens?
If 99 percent of the samples are no click, and a classifier always outputs no click, what accuracy does it reach?
Reveal answer
About 99 percent, the same as the share of no-click samples.
Accuracy counts the fraction of predictions that match the true labels. When almost every true label is no click, a constant no-click answer matches almost every label, even though the model never identifies a single click.
Tracing the 9:1 Digit Experiment
The same structure appears in a programming experiment. The digits dataset is turned into a 9:1 imbalanced problem by asking whether each image is the digit 9 against the nine other digit classes. The positive class is the digit 9, and the negative class contains everything else. Because the negative class is far more common, a model that ignores the images entirely and always answers with the common class already looks reasonable on paper.
Before running the experiment, make a prediction. The code uses DummyClassifier with the strategy set to most_frequent. This classifier does not look at the input features in any useful way. It learns which class is most common in the training labels and then predicts that class for every test point. The program below is written to match that description. The imports for the split and for numpy are included so the program runs as written.
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score
import numpy as np
digits = load_digits()
X = digits.data
y = (digits.target == 9)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train, y_train)
pred_most_frequent = dummy.predict(X_test)
print(np.unique(pred_most_frequent))
print(accuracy_score(y_test, pred_most_frequent))- 1Lines 1 to 5 import the tools. The split and numpy imports do not change the experiment.
- 2The line that builds y is the first candidate location for trouble. The digit 9 becomes True and every other digit becomes False, so the label distribution is set here.
- 3The split gives X_train and
y_train, which the baseline will learn from, and X_test andy_test, which are held back for scoring. - 4The fit step is the second candidate location. The DummyClassifier records which class dominates the training labels. It never uses the pixel features to decide anything.
- 5The predict step is the third candidate location. Every test point receives the same class, so
pred_most_frequentis a constant array. - 6The unique-labels line exposes the divergence. If only one label appears, the model is not discriminating between classes.
- 7The final score is only a summary of the steps above. It reports how often the constant predictions matched the true labels.
- Load the digit images and their labels
- Build the labels: True for digit 9, False for everything else
- Split into training and test sets
- Fit the baseline on the training labels only
- Predict a label for every test image
- Diagnostic: list the distinct predicted labels
- Summary score: fraction of test predictions that match the truth
What do you think happens?
What does the unique-labels line print when the most frequent training class is False?
Reveal answer
A single label, False, because every test prediction is the common class.
Since the negative class is the majority, the baseline predicts False for every test point, so the set of unique predicted labels contains only one value. The accuracy then mostly reflects how often the test labels are False.
Locating the Step That Misleads
When a score surprises you, the useful question is where the result stopped matching your expectation. In the digit program there are three candidate locations: the label construction, the fit step, and the prediction step. The printed score comes last, and it only summarizes what happened earlier. Tracing the variables in order (y, the splits, the fitted model, the predictions, then the score) lets you see exactly which step produced the constant output.
The diagnostic that exposes the divergence is the unique-labels line. If the predictions contain only one label, the model is not discriminating between classes, and the accuracy number is measuring the label distribution rather than model quality. Checking the set of predicted labels is cheap and often more revealing than the score itself.
Majority-Class and Random Baselines
Majority-class and random baselines show what a score would be without real learning. A majority-class baseline, such as the most_frequent strategy above, always predicts the common class. A random baseline makes random predictions. A useful model should clearly beat both of them on the same data.
The default DummyClassifier is a random baseline. It produces the classes in the same proportions as the training set. Because the rare class is rare in the training data, the random predictor still outputs the common class most of the time. This is why its accuracy stays high even though it has no skill.
The results rank the models in an intuitive order. The random dummy is the worst by accuracy, and LogisticRegression is very good. Yet the random classifier still reaches over 80 percent accuracy. A score that a random guesser can reach is not a score that tells you much about skill. The practical test is to compare any model's score against both baselines before deciding it is meaningful.
Judging a Small Gap Over the Baseline
Next, the same evaluation is applied to a decision tree limited to a maximum depth of 2. The tree uses the features, so it is a genuine learned model. Its accuracy is only slightly higher than the constant predictor's. According to accuracy alone, the tree barely beats doing nothing.
The source names two possible explanations. Either something is wrong with how the decision tree was used, or accuracy is in fact not a good measure for this problem. A small gap between a model and a baseline is ambiguous. You cannot resolve that ambiguity by staring at the accuracy value. You need a measure that is sensitive to the rare class, and you need to check your procedure before drawing a conclusion.
The source is explicit about what this means for metric choice. Accuracy is an inadequate measure for quantifying predictive performance in this imbalanced setting. Better metrics are expected to say how much better a model is than most-frequent or random predictions. A metric that cannot separate those two trivial predictions from real models cannot be trusted to rank real models. This article does not introduce those alternative metrics; it establishes the test they must pass.
Mistakes Beginners Make With Accuracy
Common mistake Treating a high accuracy value as proof that a model is good
What it looks likeA classifier reports 99 percent accuracy on a dataset where 99 percent of samples are no click.
Why it failsThe constant no-click model reaches the same 99 percent without learning anything about users or items.
Do this insteadCompare the score against a majority-class baseline and a random baseline on the same data before trusting it.
Common mistake Reading the printed score without checking what the model predicted
What it looks likeThe program prints an accuracy value but nobody looks at the unique predicted labels.
Why it failsIf every prediction is the same label, the score is measuring the label distribution, not model quality.
Do this insteadPrint the unique predicted labels as a cheap diagnostic before interpreting the score.
Common mistake Assuming a small gap over the baseline is a clear win
What it looks likeA depth-2 tree scores only slightly above the constant predictor, and the result is declared a success.
Why it failsA small gap is ambiguous. It may reflect a flawed procedure or a metric that is not sensitive to the rare class.
Do this insteadCheck the procedure first and use a measure that responds to the rare class.
Common mistake Forgetting that the random baseline also reaches a high score on imbalanced data
Checking a Score Before You Trust It
A teammate reports a model with 95 percent accuracy on a dataset where the rare class is about 5 percent of the samples. Using only the ideas in this article, describe the two baselines you would compute first, what each one would score, and what result would convince you the model has learned something useful.
Hints
- Think about what a constant predictor of the common class would reach on this data.
- A useful model should clearly beat both the majority-class baseline and the random baseline.
- Consider which printed value you would inspect to see whether the predictions vary.
Key Takeaways
- A dataset is imbalanced when one class is much more frequent than the other, and this is the normal situation in many real problems such as click-through prediction.
- Accuracy can reach 99 percent by always predicting the common class, so a high accuracy value alone cannot prove a model is good.
- Majority-class and random baselines show what a score would be without real learning, and a useful model should clearly beat both on the same data.
- When a score surprises you, trace the pipeline step by step and check the unique predicted labels to locate where the result diverges from expectation.
- Before relying on a score for an imbalanced problem, ask whether it separates trivial baselines from the model you actually care about.