Classifier Uncertainty Estimates

A fitted classifier offers more than a predict method. It also offers decision_function, which returns a number for every sample you pass in. Predict answers the question of which class a sample belongs to. Decision_function answers a more detailed question: how strongly, and in which direction, the model leans for each sample.

In the binary classification case, decision_function returns an array whose shape is $(n_{\text{samples}},)$. Each entry is a single floating-point number describing one sample. The source example makes this concrete: the test input has shape $(25, 2)$, meaning 25 samples with 2 features each, while the decision function output has shape $(25,)$. The feature dimension has disappeared because the classifier has collapsed each sample's features into one score.

The diagram shows the first six scores from the source run, one per sample, stacked as a single column. Each sample contributes exactly one number, regardless of how many features it started with. The matrix of inputs with many columns becomes a column vector of scores, and that is why the output shape has only one dimension.

Each score measures how strongly the model believes a point belongs to the positive class, which in this example is class 1. A large positive score means the model is confident the point belongs to the positive class. A negative score means the model leans toward the negative class, which is the other class in the problem. The sign gives the direction of the lean, and the magnitude gives its strength.

The sign alone reproduces the predicted labels. If you compare each score with 0, you get a boolean array that is True for positive scores and False for negative scores. In the source example, that boolean array is True for the samples that predict returns as red and False for the samples that predict returns as blue. The thresholded decision function and the predictions therefore agree on which samples belong to the positive class. The source's rule is a comparison with 0; it does not describe a separate rule for a score of exactly 0, so this article does not assume one.

The boolean array tells you the decision for each sample, but it does not carry the original label names. It only says True or False. To get the labels that predict returns, you need one more piece of information: which label corresponds to False and which corresponds to True. The classes_ attribute provides that mapping.

For binary classification, the negative class is always the first entry of classes_, and the positive class is always the second entry. The procedure in the source converts the True and False values into 1 and 0, then uses those integers as indices into classes_. Indexing with 0 selects the negative class, and indexing with 1 selects the positive class. When the rebuilt array is compared with the output of predict across all samples, the check reports True only if every element matches. In the source run the result is True, which confirms that the sign of the decision function, combined with classes_, reproduces predict exactly.

The numeric range of decision_function is not fixed. It can be arbitrary, and it depends on both the data and the model parameters. The source run reports a minimum of -7.69 and a maximum of 4.29 on the test set. A different dataset or different model settings could produce a different spread, so a score of 2 from one model does not necessarily mean the same thing as a score of 2 from another.

decision_function gives one floating-point score per sample in the binary case, so the output shape is $(n_{\text{samples}},)$. The sign of each score gives the direction of the model's lean, and its magnitude gives the strength. The classes_ attribute, with index 0 for the negative class and index 1 for the positive class, turns those signs back into the labels that predict returns. Because the numeric range is arbitrary and depends on data and parameters, read the sign for the decision and the relative ordering within one model, not the raw value, as a confidence level.