Comparing Baseline and Trained Classifiers

Suppose a classifier reports an accuracy score and you are asked whether it is good. A single number can tell you how often the model was right overall, but it cannot tell you which kinds of mistakes it makes. Two models with similar accuracy can fail in very different ways, and one of them may be missing the cases that matter most. A confusion matrix opens that single number up so you can see the individual outcomes that produced it.

In this article we start from a real evaluation. A logistic regression model has already produced predictions on a held-out test set, and those predictions are stored in a variable named pred_logreg. The true labels for the same test samples are in y_test. Passing those two arrays to confusion_matrix turns them into a grid of counts. Before reading the grid, it helps to know what it is organized around, so we will work through that next.

The orientation of the matrix is the key to reading it. Each row is tied to the true class of a group of samples, and each column is tied to the class the model assigned to them. Every entry is a count of how many samples with a given true class received a given predicted class. The first row therefore describes everything that truly belonged to the not-nine group, split by what the model guessed. The second row describes everything that truly belonged to the nine group, split the same way.

The diagram above places the four cells in their matrix positions. The top row is the true not-nine class and the bottom row is the true nine class. The left column is the predicted not-nine class and the right column is the predicted nine class. Once you hold that grid in mind, reading any count is just a question of asking which true class and which predicted class meet in a given cell.

This layout gives the matrix a useful property. The two cells on the main diagonal, running from top-left to bottom-right, are the cases where the true class and the predicted class agree. Those are the correct classifications. The two off-diagonal cells are the mistakes, and each one names both the class a sample really belonged to and the class the model wrongly chose for it.

Binary classification becomes easier to discuss once one class is designated as the positive class. In the nine-versus-rest task, declaring a nine to be the positive class lets the four cells receive standard names. The names describe whether the model was right or wrong, and which class it predicted.

Notice that each name is built from two independent judgments. The first word says whether the prediction was correct, true or false. The second word says what the model output, positive or negative. A false negative is therefore a mistake in which the model output negative, and the mistake is that the sample was actually positive. A false positive is the reverse: the model output positive, and the sample was actually negative.

Now consider the question the matrix is meant to answer: where did the model's output diverge from the true labels? Suppose you want to know how many nine-images the classifier missed entirely. You do not need to recount the test set. You only need the row for true nine samples and the column for predicted not-nine samples, which is the bottom-left cell. That cell is the false negative count, and it is the one location that directly answers the question.

Comparing models side by side is where the matrix becomes most valuable. In the source comparison, a confusion matrix was computed for each of four fitted models: two dummy baselines, a decision tree, and the logistic regression. Each call used the same true labels, so the matrices could be read against a common reference. Reading across the results exposed behavior that a single score would not hide.

The most frequent class baseline never produces a positive prediction, so its true positive count is zero and its 47 positive samples all land in the false negative cell. The other dummy model does predict positives, but it produces only 4 true positives against 42 false positives, so most of its positive predictions are wrong. The decision tree is much more sensible, with 23 true positives. Logistic regression is better than the tree on every count, with more true positives and true negatives and fewer false positives and false negatives.

Inspecting full grids is enough to tell which model behaves reasonably in a small comparison like this one. Doing the same inspection for many models and many test sets is slow, which is why the outcomes are later condensed into summary measures. Accuracy is the simplest of those summaries.

The matrix gives a direct route to accuracy. Accuracy is the number of correct predictions divided by the number of all samples. In terms of the four outcomes, the correct predictions are the true positives and true negatives, which are the diagonal cells. The number of all samples is the sum of every cell in the matrix. Equivalently, accuracy is the diagonal sum divided by the sum of all entries.