Confusion Matrix and Classification Reports

When a classifier labels a sample as positive or negative, it feels as though the model simply answers the question. Underneath, the model usually computes a score for each sample and then compares that score against a fixed cutoff. That cutoff is the decision threshold. Because it is chosen once and then applied silently, it is easy to forget that it exists at all, and easy to assume it is the only correct place to draw the line.

This article starts from that hidden assumption and tests it. The confusion matrix and the classification report describe the predictions a model made. But those predictions have already discarded information about how confident the model was. Recovering that information, and seeing what happens when the cutoff moves, is the subject of this lesson.

Think of the decision_function output as a score along a line. Every sample has one score. The cutoff splits that line into two regions, and any sample scoring above the cutoff is assigned to class 1. Everything at or below the cutoff falls into the class 0 region. The diagram below traces five samples through this process.

The scores in this diagram are illustrative values chosen to show the rule, not results from the source. The rule itself is the point: the predicted class is decided entirely by which side of the cutoff a score lands on.

Models with a predict_proba method follow the same logic on a different scale. Their outputs fall from 0 to 1 and can be read as probabilities. The default threshold of 0.5 means a sample is labeled positive when the model is more than 50 percent sure it belongs to the positive class. Raising the threshold requires the model to be more confident before it makes a positive decision, which makes negative decisions easier to reach.

Consider an imbalanced problem with many negative samples and few positive ones. The classifier here is a kernel SVM, and its default cutoff for class 1 is a decision_function value greater than 0. Suppose your application values catching positive cases more than avoiding false alarms, as in a cancer screening setting. You want to change the cutoff so that more samples are flagged as class 1. Before you look at any results, make a prediction about what happens to the two measures you care about.

A confusion matrix sorts every test sample into one of four cells by comparing its true class with its predicted class. A true positive is a class 1 sample predicted as class 1. A false positive is a class 0 sample predicted as class 1. A false negative is a class 1 sample predicted as class 0. A true negative is a class 0 sample predicted as class 0. Applying a threshold is the step that fills these cells, because the predicted class is what the threshold produces.

The reported figures constrain the counts. Class 1 recall of 0.67 on 9 positive samples means about 6 are found and 3 are missed. Class 1 precision of 0.35 with 6 true positives implies roughly 11 false positives. The table below shows one set of counts consistent with the reported report at the default cutoff. It is a reconstruction for illustration, not a count printed in the source.

Precision divides true positives by all predicted positives, so every false alarm lowers it. Recall divides true positives by all actual positives, so every missed positive lowers it. Both depend on the same confusion matrix cells, which is why changing the threshold moves them together through the cell counts.

Now lower the cutoff for class 1 from 0 to -0.8 and evaluate on the same test set. The class 1 rows should be read first, since they are the ones the change was meant to improve. Then check whether the overall averages moved in the same direction, because a single class can hide what happens to the other.

In the source example, lowering the cutoff from 0 to -0.8 raised class 1 recall from 0.67 to 1.00 and lowered class 1 precision from 0.35 to 0.32. Recall went up because every true class 1 sample is now caught. Precision went down because the extra positive predictions include more false alarms. This is the general pattern: lowering the threshold for the positive class tends to raise recall and lower precision, and raising it tends to do the opposite.

The reason is geometric. Moving the cutoff leftward enlarges the class 1 region of the score line, so more samples land in it. The source describes this as classifying a larger region of space as class 1, which is visible in the heat map of the decision function.

The same move can be traced through the cells. Under the illustrative reconstruction, a cutoff of -0.8 that captures all 9 class 1 samples with a precision of 0.32 implies about 28 predicted positives, so about 19 are false positives. That leaves roughly 85 true negatives out of 104 class 0 samples. These counts are a consistent illustration of the reported figures, and the direction of change is what matters: false negatives fell from 3 to 0, while false positives rose.

Once the cutoff is understood as a tunable choice, the next question is how to tune it. It is tempting to scan several thresholds on the test set and keep the one that looks best. That is a mistake. Setting a decision threshold on the test set is likely to yield overly optimistic results, in the same way that tuning any other parameter on the test set would.

Use a validation set or cross-validation to compare candidate thresholds. Choose the one that fits your priorities on that held-out data, then evaluate once on the test set. The test set stays untouched during selection, so its final report remains an honest estimate. The source gives no rule of thumb for the exact threshold value, because the range of the decision_function can be arbitrary. The threshold has to be found empirically for the model and data at hand.