Imbalanced Datasets and Baseline Models

Suppose a dataset has 100 samples, and only 10 of them belong to the class you care about. This class is the minority class. The other 90 samples belong to the majority class. A classifier that never looks at the input and simply answers with the majority class will still be right 90 times out of 100. Its accuracy is 90 percent, which sounds impressive, yet it has found none of the minority cases at all.

This is the central problem of imbalanced datasets. A single accuracy value counts correct predictions, both true positives and true negatives, and divides by the total number of samples. Two models can share the same accuracy while one floods the output with false alarms and the other quietly misses most of the real positive cases. Precision and recall were developed to separate those two failure modes, and the F-score then combines them into one value.

A confusion matrix records how a binary classifier's predictions line up with the true labels. Its cells count four outcomes. A true positive is a sample that is really positive and was predicted positive. A true negative is a sample that is really negative and was predicted negative. A false positive is a sample that is really negative but was predicted positive, which is a false alarm. A false negative is a sample that is really positive but was predicted negative, which is a miss.

Precision and recall both use the true positives in their numerator. They differ only in their denominators, and that single difference determines which mistake each metric counts against the model. The diagram below shows which outcomes feed into each metric.

Precision answers one question: of all the samples the model labeled as positive, how many really are positive? It only looks at the positive predictions. It ignores the negatives the model correctly rejected, and it ignores the positives the model missed entirely. Precision is therefore sensitive to false positives.

Recall flips the viewpoint. Instead of asking how many of the positive predictions are right, it asks how many of the actual positive samples the model managed to capture. Its denominator is the set of true positives plus the positives that were missed, which are the false negatives. Recall is therefore sensitive to false negatives.

Recall is the right yardstick when missing a positive case is the costly error. In cancer diagnosis, it is important to find all the people who are sick, even if a few healthy patients are flagged along the way. A missed sick patient is a false negative, and high recall is the property that keeps those rare.

The two metrics pull against each other. Changing how strict a model is about predicting positive moves a sample across the boundary between positive and negative predictions. Making the model more permissive usually adds true positives and false positives together, which tends to raise recall and lower precision. Making it stricter usually removes false positives but also removes some true positives, which tends to raise precision and lower recall. The two extremes below show why neither can be used alone.

Because precision and recall each tell only part of the story, it helps to have a single summary that accounts for both. The F-score is the harmonic mean of precision and recall. The most common variant weights the two equally and is called the F1-score.

A harmonic mean differs from the ordinary average. Because it is built from reciprocals, the result stays close to the smaller of the two inputs. Take precision 0.9 and recall 0.1. The ordinary average is 0.5, which looks comfortable. The F1-score is 2 times 0.9 times 0.1 divided by 1.0, which is 0.18. The low recall drags the score down rather than being hidden behind the high precision. That is why the F-score is considered a more faithful picture of a good model than accuracy on imbalanced binary datasets.

The source applies these metrics to a 'nine vs. rest' dataset, where the class 'nine' is the positive class and therefore the minority class. Four sets of predictions were produced: from a most-frequent baseline, a dummy classifier, a decision tree, and a logistic regression. Each set is scored against the same true labels. The ordering you would expect appears in the result: the logistic regression scores highest and the most-frequent baseline scores lowest.

The baseline's zero score is a warning sign rather than a broken metric. The baseline makes no positive predictions at all, so its precision denominator is zero and the ratio cannot be computed in the usual way. In the source, this zero denominator is the reason an error message appears for that model. A zero F1 here means the model made no positive predictions.

The F1-score also separates models that accuracy alone does not distinguish sharply. The gap between the dummy classifier and the decision tree is clearer when read through F1 than through accuracy. This is the practical reason the F-score is often chosen for imbalanced problems.

When you want precision, recall, and F1-score together for every class, scikit-learn provides the classification_report convenience function. It prints one row per class and a final row of averages. Each row reads the metrics with that class treated as the positive class. The columns are precision, recall, F1-score, and support. The rightmost column, support, is the count of samples in that class according to the ground truth. The bottom row is an average of the per-class values weighted by support.

Read the report row by row and column by column. A row shows how one model behaves on one class. A column shows how every class behaves on one metric. The most-frequent baseline has a recall of zero for the 'nine' class because it never predicts that class. To locate where one model's results diverge from what you expected, compare a column across models, then look at the matching row for the other class.

Precision, recall, and F1 are defined relative to whichever class you call positive, so swapping the label changes every number. In the report for the most-frequent baseline, treating 'not nine' as the positive class gives an F1-score of about 0.94. The reason is that this baseline predicts 'not nine' for every sample, so it recalls all of the 'not nine' samples. The same model looks very different depending on the choice.