Imbalanced Binary Classification

Many learners treat the output of a classifier as if the final label were fixed by the model itself. You call predict, a label comes back, and it feels as though the model has settled the question. This section challenges that belief. The label you receive is the result of applying a cutoff to a score, and that cutoff is a choice that can be moved. Moving it changes which samples count as positive, which changes precision and recall, and sometimes changes the result you were hoping for.

The setting for this article is an imbalanced binary task. The negative class is much larger than the positive class, and the split sizes make that visible. The training portion has 400 negative points and 50 positive points. The test portion has 104 negative samples and 9 positive samples. Because the classes are so uneven, a classifier can spend most of its effort on the large class, and the numbers in a classification report reflect that skew.

A classifier that exposes decision_function produces a degree of certainty for each sample, not just a yes or no. Each sample receives a score. Turning that score into a label requires a cutoff. For binary classification the default cutoff is 0 for decision_function. A sample whose score sits above the cutoff is labeled as the positive class, and a sample below it is labeled as the negative class.

Follow the illustrative scores below through the default cutoff. Each point is compared with the cutoff, and the side it lands on decides its label. The scores here are invented to show the mechanism; they are not taken from the source's model.

The same rule applies to every sample. A score above the cutoff produces the positive label, and a score below it produces the negative label. Nothing about the model's fitted parameters is consulted at this step. Only the score and the cutoff are compared.

Probability outputs are often easier to reason about. Their values lie on a fixed scale from 0 to 1, and they are meant to represent the model's probability for the positive class. With the default threshold of 0.5, a point is classified as positive when the model is more than 50 percent confident. Raising the threshold means the model must be more confident before it makes a positive decision. It also means the model can be less confident before making a negative one.

Before looking at any numbers, commit to a prediction. Suppose the positive class matters a great deal, and you want the model to find as many positive samples as possible, even if it occasionally flags a negative sample by mistake. You lower the decision_function cutoff from 0 to a value near -0.8, so more points count as positive. Think about what should happen to recall for the positive class, and what should happen to precision.

Now trace what happens to individual points. The illustrative sample below has a score of -0.5. Under the default cutoff of 0, it sits below the line and is labeled class 0. Under the lowered cutoff of -0.8, it sits above the line and is labeled class 1. Its score did not change, and the model was not retrained. Only the cutoff moved.

Every point that lies between the two cutoffs changes label, and every point outside that band keeps its label. Points that move from class 0 to class 1 can be correct or incorrect, depending on their true labels. Correct moves raise the true positive count. Incorrect moves raise the false positive count. Recall for class 1 grows when the first kind of move dominates, and precision for class 1 drops when the second kind is common.

A classification report hides its counts behind a few decimals, but the counts are what you need when you want to understand a failure. TP counts true positives, FP counts false positives, and FN counts false negatives. Precision asks how many of the samples labeled positive really are positive. Recall asks how many of the truly positive samples were found.

Both formulas share the true positive count in the numerator, which is why precision and recall move together in some situations and pull apart in others once the threshold changes. The denominator of precision collects everything labeled positive. The denominator of recall collects everything that is truly positive. To recover the counts from a report, work backward. If you know the precision and the number of predicted positives, TP is their product. FN then follows from the number of truly positive samples minus TP.

Read the per-class rows of the report rather than only its averages. The row for class 1 is where a divergence from what you wanted usually appears. Its recall may be small while its precision is mixed, and the overall average can still look respectable because class 0 dominates it. Class 0 often shows the mirror image: when class 1 recall climbs, class 0 precision tends to rise and class 0 recall tends to fall.

Once you see that the threshold changes the outcome, it is tempting to try many values and keep the one that looks best. The question is where you do that search. The threshold should be chosen on a validation set or with cross-validation. The test portion is held back so that the final model is evaluated on data it has not been used to shape. Searching over the test portion lets its information leak into the decision, and the resulting numbers look better than the model will perform on new data.