Accuracy as a Classification Metric

Accuracy gives you one number summarizing how often a classifier is right. That number is useful, but it compresses everything the model did into a single value. Two classifiers can earn the same accuracy while making very different kinds of mistakes, and the single number cannot tell them apart. A confusion matrix opens that number up so you can see which kinds of mistakes a classifier makes.

We start from a realistic situation. A logistic regression model has already produced predictions on a held-out test set, and those predictions are stored in a variable named pred_logreg. The true labels for the same test samples are in y_test. Passing those two arrays to confusion_matrix turns them into a matrix of counts.

The output is a grid of four numbers. Before reading further, try to guess what the grid is organized around. Which two pieces of information is it counting against each other? Keep your guess in mind, because the next section answers it directly.

The orientation of the matrix is the key to reading it. Each row is tied to the true class of a group of samples, and each column is tied to the class the model assigned to them. Every entry is a count of how many samples with a given true class received a given predicted class. The first row therefore describes everything that truly belonged to the not-nine group, split by what the model guessed. The second row describes everything that truly belonged to the nine group, split the same way.

The diagram places the four cells in their matrix positions. The top row is the not-nine true class and the bottom row is the nine true class. The left column is the not-nine prediction and the right column is the nine prediction. Once you hold that grid in mind, reading the counts becomes a single question: which true class and which predicted class meet in a given cell?

This layout gives the matrix a useful property. The two cells on the main diagonal, running from top-left to bottom-right, are the cases where the true class and the predicted class agree. Those are the correct classifications. The two off-diagonal cells are the mistakes, and each one names both the class a sample really belonged to and the class the model wrongly chose for it.

Binary classification is easier to discuss once one class is designated as the positive class. In the nine-versus-rest task, declaring a nine the positive class lets the four cells receive standard names. The names describe whether the model was right or wrong, and which class it predicted.

The names are built from two independent judgments. The first word says whether the prediction was correct, true or false. The second word says what the model output, positive or negative. A false negative is therefore a mistake in which the model said negative, and the mistake is that the sample was actually positive. Read this way, the two correct cells are the true positives and true negatives on the diagonal, and the two mistake cells are the false positives and false negatives off the diagonal.

Now consider the question the matrix is meant to answer: where does the model's output diverge from the true labels? Suppose you want to know how many nine-images the classifier missed entirely. You do not need to recount the test set. You need only the row for true nine samples and the column for predicted not-nine samples, which is the bottom-left cell. That cell is a false negative count, and it is the single place where a nine was reported as something else.

The worked example shows a useful habit. Each off-diagonal cell answers one specific question. The top-right cell counts the not-nines the model called nines, and the bottom-left cell counts the nines the model called not-nines. Before trusting any summary, check which of these cells is largest, because the largest mistake often matters most for the task.

The matrix gives you a direct route to accuracy. Accuracy is the number of correct predictions divided by the number of all samples. In terms of the four outcomes, the correct predictions are the true positives and true negatives, and the number of all samples is the sum of every cell in the matrix. The correct predictions sit on the main diagonal, so accuracy is the diagonal sum divided by the total of all entries.

The matrix becomes most valuable when you compare models side by side. A useful comparison computes a confusion matrix for several fitted models using the same true labels, so each matrix can be read against a common reference. The source example compares two dummy baselines, a decision tree, and a logistic regression.

Reading across those matrices exposes behavior that a single score would not. The most frequent class model never produces a positive prediction, so its true positive count is zero and all 47 of its positive samples become false negatives. The other dummy model does predict positives, but it produces only 4 true positives against 42 false positives, so most of its positive predictions are wrong. The decision tree is much more sensible, with 23 true positives. Logistic regression is better than the tree on every count: more true positives and true negatives, and fewer false positives and false negatives.