Introduction to Classifier Uncertainty

A classifier that predicts one of several classes does not have to reveal only its final answer. Behind each prediction it has scored every possible class, and some classifiers expose those internal scores through a method called decision_function. Instead of asking only what the classifier predicts, this article asks what the classifier is weighing when it makes that prediction.

The running setting is a gradient boosted classifier trained on the Iris dataset, which has three classes. After training, we ask this classifier for its decision function on the test set. Before reading the numbers, predict what kind of output you expect: one number per data point, one number per class, or something else.

The decision function for multiclass classification has shape $(n_{\text{samples}}, n_{\text{classes}})$. Each row corresponds to one data point, and each column corresponds to one class. The value stored in a given cell is a certainty score for that class on that data point.

Column order follows the class labels the classifier learned. Column 0 holds the score for class 0, column 1 for class 1, and column 2 for class 2. The number of columns tells you how many classes exist, and the column positions tell you which label each score belongs to. Keep this mapping in mind whenever you debug, because a score in the wrong column is a labeling error, not a model error.

A larger certainty score means a class is more likely, and a smaller score means it is less likely. To interpret a data point, read its row one at a time and compare the scores across its columns. The class whose column holds the largest value is the one the classifier considers most likely for that point.

The example row above is a made-up row, chosen only to practice reading scores. It is not output from a real model. Class 2 has the largest value, so it is the class this row favors. Notice that the rule is comparative. The raw values -1.2 and 0.4 are not wrong or invalid simply because they are below some cutoff; they are simply smaller than 2.1 in the same row.

Because the largest entry in each row marks the most likely class, predictions can be recovered directly from the decision function. You find the index of the maximum entry for each data point. The NumPy call np.argmax(scores, axis=1) does this. It returns, for every row, the column index where that row's maximum sits.

The source runs this same kind of check: it prints the argmax of the decision function and the output of gbrt.predict on the same test set, and the two arrays match entry for entry. For this classifier, then, the predicted class is the column with the maximum certainty score in each row. The mapping from column index back to class label works here because the classes are labeled 0, 1, and 2 in the order the classifier learned them.

Another method, predict_proba, also returns one value per class for each data point. Its output has the same shape as the decision function, $(n_{\text{samples}}, n_{\text{classes}})$, so the two outputs are laid out the same way. They are not the same quantity, though. The decision function gives certainty scores, while predict_proba gives probabilities whose values for each data point sum to 1. The source does not state that the decision function's values sum to 1, so do not assume it.

The decision function followed by argmax is a two-step pipeline, so you can check each step separately when a prediction looks wrong. First confirm the shape of the decision function. Then confirm that the argmax runs across axis 1. Finally, compare the recovered predictions with gbrt.predict on the same data. If the two arrays disagree, the divergence lies in the argmax step or in how the rows were prepared, not in the classifier itself.