Uncertainty Estimates From Classifiers
For a multiclass classifier, decision_function returns an array of shape (n_samples, n_classes), where each column gives a certainty score for one class.
Asking a Classifier How Sure It Is
Once a classifier has been trained on labeled examples, it can be used on new data. A common first use is to ask which class a data point belongs to. But a single label hides a lot. Two samples might both be labeled as the same class even though the model was confident about one of them and nearly torn about the other. This article looks at the richer output that a trained classifier can give you, so that the label you see can be explained rather than just accepted.
What do you think happens?
One test sample receives three certainty scores, one for each class, and the middle score is clearly the largest. Which class would the model report for that sample if it predicted from those scores alone?
Reveal answer
The middle class, because its score is the largest
The predicted class for each sample is the column with the maximum entry in that sample's row. A larger score for a class means that class is more likely for that sample, so the largest of the three scores identifies the class the model favors.
Two Questions a Trained Classifier Can Answer
After a classifier is trained, you usually ask it one of two questions. The first is which class this data point belongs to. The predict method answers that question with a single label for each sample. The second is how strongly the model favors each class. The decision_function method answers that second question with a set of raw scores, one score for each class.
This article focuses on the second output. It contains more information than the final label alone, and understanding it makes the predicted label easier to explain. If you can read the scores, you can point to the row of numbers that produced any label the model reports.
The running example in this material comes from the Iris dataset, which contains three classes of flower. The data is split into training and test portions, and a GradientBoostingClassifier is fitted on the training portion. Calling decision_function on the test portion then produces the scores examined here.
The Shape of decision_function Output
For a multiclass classifier, decision_function returns an array of shape (n_samples, n_classes). Each row corresponds to one data point, and each column corresponds to one class. A single cell therefore holds one score: the certainty score that the model assigns to one class for one sample.
The two axes of the array answer different questions. Moving along the first axis, from one row to the next, moves from one sample to another. Moving along the second axis, from one column to the next, moves from one class to another. Reading an individual value means naming both: which sample, and which class.
The diagram above uses a small two-sample, three-class array built for illustration. The highlighted cells mark the largest score in each row, which is the idea the next sections develop. Each row is one sample's complete set of certainty scores, and each column is always the same class across all samples.
Reading Scores Within Each Row
A larger score means the class is more likely for that sample. Because of this, the scores are compared within each row rather than read as absolute values. The question for any one sample is which of its scores is largest, not whether a particular number is large in some universal sense.
The scores are not restricted to a fixed range, and they do not need to add up to any particular total. In the Iris test output, several rows show one column standing well above the others, with the remaining values negative or near zero. This is a decisive prediction. In other rows, the top two columns are closer together, which signals that the model is less decisive about that sample.
Recovering the Predicted Label with argmax
The predicted class can be recovered directly from the decision scores. For each data point, the predicted class is the column with the maximum entry. NumPy's argmax function returns the index of that maximum, and its axis argument tells it which direction to search. To search across the columns for each row, you use axis=1, because the classes lie along the second dimension of the array.
[1 2 1]- scores has one row per sample and one column per class
- axis=1 searches across the columns, so each row is compared with itself
- The result holds one column index per sample; those indices are the class positions
Predicting the class for each row
Given the three-sample score array from the code example, find the predicted class index for each sample using argmax along the class axis.
- 1Identify the axis
The classes lie along the second dimension, so the search runs across columns with axis=1.
- 2Scan row 0
The values are -1.2, 2.3, and 0.1. The largest is 2.3, at column index 1.
- 3Scan row 1
The values are 0.4, -0.5, and 1.8. The largest is 1.8, at column index 2.
- 4Scan row 2
The values are 0.9, 1.1, and -2.0. The largest is 1.1, at column index 1.
- 5Collect the results
Stacking the three indices gives one predicted class per sample, in the same order as the rows.
The predicted class indices are [1, 2, 1], which matches the printed output of the argmax call.
In the source example, the array produced by argmax over the decision scores is identical, index by index, to the array returned by predict on the same test data. The predicted class is therefore not a separate computation hidden inside predict. It is the argmax of the same scores that decision_function exposes. This lets you explain any predicted label by pointing to the row of scores that produced it.
decision_function and predict_proba Side by Side
The output of predict_proba has the same shape as the output of decision_function: one row per sample and one column per class. The shape alone therefore does not tell you which method produced an array. What differs is the meaning of the numbers.
| Property | decision_function | predict_proba |
|---|---|---|
| Shape | (n_samples, n_classes) | (n_samples, n_classes) |
| Rows represent | one sample each | one sample each |
| Columns represent | one class each | one class each |
| Meaning of values | certainty scores with relative ordering | probabilities |
| Row sums | not constrained to sum to 1 | sum to 1 |
| Predicted class | column with the maximum entry | column with the maximum entry |
Both arrays share a layout, but only one of them can be read as a distribution over classes.
Because the two arrays share their layout, you can apply the same argmax step to either one. The label that results is determined by the largest value in each row. The difference is that predict_proba values can also be read as probabilities that sum to 1, whereas decision scores can only be compared against each other.
Mistakes When Reading Decision Scores
Common mistake Treating a decision score as a probability
What it looks likeReading a score of 2.3 as if it meant a 230 percent chance of the class.
Why it failsDecision scores carry only relative ordering. They are not constrained to a range and do not need to add up to any total.
Do this insteadCompare the scores within a row and identify the largest one. Use predict_proba if you need values that can be read as a distribution.
Common mistake Searching down the rows instead of across the columns
What it looks likeCalling np.argmax(scores, axis=0) and expecting one predicted class per sample.
Why it failsThe classes lie along the second dimension, so axis=0 compares samples across each class column instead of comparing classes within a sample.
Do this insteadUse axis=1 to find the maximum within each row.
Common mistake Assuming the same shape means the same meaning
What it looks likeAssuming that two arrays with shape (n_samples, n_classes) must hold the same kind of numbers.
Why it failsThe shape of
decision_functionandpredict_probaoutput is identical, but the meaning of their values differs.Do this insteadCheck which method produced the array before interpreting the values.
Common mistake Forgetting that the index from argmax is a column position, not a class name until mapped
Checking Your Reading of a Score Matrix
A test sample receives decision scores of 0.31, 0.29, and -1.0 for three classes. Which class does argmax select for this sample, and how decisive is the model about it? Explain your answer using the phrase relative ordering.
Hints
- Find the largest of the three values and note its column position.
- Compare the top two values with each other before deciding how decisive the model is.
- Remember that the scores are not probabilities, so do not check whether they sum to 1.
Summary of Reading Decision Scores
- 1
decision_functionreturns an array of shape (n_samples,n_classes): rows are samples, columns are classes, and each cell is a certainty score for one class on one sample. - 2Larger scores mean a class is more likely for that sample, so the scores are compared within each row rather than read as absolute values.
- 3The predicted class is the column with the maximum entry, found with
np.argmaxalong axis=1, and it matches the output of predict on the same data. - 4
predict_probahas the same shape asdecision_function, but its rows are probabilities that sum to 1, while decision scores carry only relative ordering.
At a glance
decision_functionreturns an array of shape (n_samples,n_classes), where rows are samples and columns are classes.- A larger certainty score means a class is more likely for that sample, so scores are compared within each row.
- argmax with axis=1 recovers the predicted class for each sample, matching the output of predict.
predict_probashares the shape ofdecision_function, but its rows are probabilities summing to 1, while decision scores carry only relative ordering.