Predicted Probabilities in Multiclass Classification

Once a classifier has been trained, you usually ask it one of two questions. The first is which class a data point belongs to. The second is how strongly the model favors each possible class. The predict method answers the first question with a single label per sample. The predicted probabilities and decision scores answer the second question with a set of numbers, one per class. This article focuses on that second kind of output, because it carries more information than the final label and because understanding it makes the predicted label easier to explain.

The running example comes from the Iris dataset, which has three classes of flower. The data is split into training and test portions, and a GradientBoostingClassifier is fitted on the training portion. The output you will read is produced by calling predict_proba on the test portion of that trained model.

For a multiclass classifier, decision_function returns an array of shape (n_samples, n_classes). Each row belongs to one sample, and each column belongs to one class. Each entry is a certainty score: the score in row i and column k says how strongly the model's output for sample i points toward class k. Because the classes lie along the second dimension of the array, the column axis is the one that indexes classes, and the row axis indexes samples.

The key to reading a decision score is that a larger value means the class is more likely for that data point. Scores are compared within a row, not read as absolute values. Within one sample's row, the column with the highest score is the class the model leans toward most. The scores are not restricted to a fixed range, and they do not need to add up to any particular total. A row with one column well above the others means the model is decisive about that sample. A row where the top two columns are close together means the model is less decisive. Negative values are normal and do not signal anything wrong by themselves; only the ordering within the row matters for the comparison.

A row of scores is informative, but often you want one label per sample. The predicted class for each data point is the column with the maximum entry in that row. NumPy's argmax function returns the index of the maximum, and its axis argument tells it which direction to search. To search across the columns for each row, use axis=1, because the classes lie along the second dimension of the array.

predict_proba returns an array of shape (n_samples, n_classes), with one row per sample and one column per class. The rows are probabilities, and for each sample they add up to 1. In mathematical terms, for sample i with K classes, the entries satisfy $\sum_{k=1}^{K} p_{ik} = 1$, so each row can be read as a distribution over the classes.

In the source run, calling predict_proba on the first six test samples produces six rows with three values each. The first row contains values near 0.107, 0.784, and 0.109 across the three columns. The sample is most strongly associated with the middle class, but the other two classes still receive some probability. Summing each row gives 1, which is the central property of this output.

The argmax gives you a column index, not a class name. A column index is meaningful only if you know which class each column stands for. A fitted classifier stores the class labels it learned in an attribute called classes_. The position of each label in classes_ corresponds to the position of the matching column in the output. To turn a winning column into a real label, index classes_ with the argmax result.

This mapping is easy to skip. When the labels are plain integers that start at 0 and run consecutively, positions and labels look identical, so it can seem unnecessary to consult classes_. The source warns that the mapping must be used especially for string labels or non-consecutive integers. In those cases, using the raw argmax as if it were a label produces wrong answers without any error message.

In the multiclass case, decision_function and predict_proba have the same shape. Both return one row per sample and one column per class. What differs is the meaning of the numbers. predict_proba values are probabilities that sum to 1 within each row. decision_function values are raw certainty scores whose only reliable information is their relative ordering within a row.