Decision Function in Binary Classification

Suppose a classifier's decision_function returns a value of 4 for one sample and a value of -3 for another. It is tempting to read these as confidence percentages, as if 4 means strong agreement and -3 means weak disagreement, with the scale running from zero to one or from zero to one hundred. This is the most common first reaction to decision_function output, and it is the misconception this article corrects.

The numbers are real-valued scores. The scale they live on is not fixed in advance. Their sign carries the decision between the two classes, while their magnitude tells you something about how far a sample sits from the separating rule. It does not give a calibrated chance that the classifier is right. The traps you will meet later in this article all grow out of the percentage reading, so keep it in mind as we work through the mechanism.

The range of decision_function is not fixed. It depends on the training data and on the model's parameters. The same classifier trained on a different dataset can produce scores that run over a completely different interval, and a score of 2 from one model does not mean the same thing as a score of 2 from another model. Because of this arbitrary scaling, the raw output is hard to interpret on its own.

The exact size of a value is useful for ranking samples relative to one another within a single model. It does not convert into a probability without a separate calibration step, and that step is outside the scope of this section. In practice this means you can say that one sample sits further from the boundary than another, but you cannot say that one sample has a 90 percent chance of belonging to its class.

To recover the predicted class, you compare every decision value with zero. Values strictly greater than zero become True, and everything else becomes False. The result is a boolean array with one entry per sample. This is the core move of the whole mechanism: the sign of the score is the only information needed to choose between the two classes, and the magnitude is discarded at this step.

A boolean is not yet a class name. For binary classification, the negative class is always the first entry of the classes_ attribute, and the positive class is the second entry. To reproduce the output of predict, you convert the booleans into the integers 0 and 1, then use those integers as indices into classes_. Index 0 selects the negative class and index 1 selects the positive class.

When your recovered labels disagree with predict, the fix is to check the stages in order and stop at the first one that diverges. For binary classification the stages are the decision values, the threshold, the boolean mask, and the classes_ lookup. Comparing your output with model.predict after each stage tells you which one broke.

In the multiclass case the decision function has the shape $(n_{\text{samples}}, n_{\text{classes}})$. Each row corresponds to one data point and each column corresponds to one class. The value in a given cell is a certainty score for that class on that data point. The column order follows the class labels the classifier learned, so column 0 is the score for class 0, column 1 for class 1, and column 2 for class 2. The shape alone tells you how many classes exist.

Read the rows one at a time. For a single data point you compare the scores across its columns, and the class whose column holds the largest value is the one the classifier considers most likely. The rule is comparative. What matters is which column is largest within a row, not whether any single number clears some fixed threshold. In the small example above, sample 0 has its largest score in column 2, and sample 1 has its largest score in column 0.

Because the largest entry in each row marks the most likely class, predictions can be recovered directly from the decision function. You find the index of the maximum entry for each data point with np.argmax(scores, axis=1), which returns, for every row, the column index where that row's maximum sits.

The predict_proba method also produces one value per class for each data point. Its output has the same shape as the decision function, $(n_{\text{samples}}, n_{\text{classes}})$, so the two arrays are laid out the same way. They are not the same quantity, and the difference is what matters when you interpret them.

Because the shapes match, it is easy to assume the numbers can be swapped. They cannot. A decision value of 2.5 does not mean the same thing as a probability of 0.5, and the decision function does not carry the sum-to-one guarantee that predict_proba does.