Decision Values in Linear Classifiers

A classifier that only reports a single label for each sample tells you the answer but hides how close the decision was. Many scikit-learn classifiers can do more. Instead of saying only which class a sample belongs to, they can report how strongly they favor each possible class. The main tool for this is the method predict_proba. This article walks through what its output looks like, how to turn that output back into a single predicted label, and why the column positions in that output must be translated into class names with care.

The output of predict_proba is an array with shape (n_samples, n_classes). Each row belongs to one sample, and each column belongs to one class. A row therefore lists, for a single sample, a probability for every class the model knows about. Because the model is choosing among several classes, the numbers in a row are meant to be read together, not one at a time.

In this formula, p_{ik} is the probability that sample i belongs to class column k, and K is the number of classes. The formula says that the probabilities in each row sum to 1. This is the central property of the output: each row is a distribution over the classes, so a larger value in one column means the model leans more toward that class, and the remaining columns share the rest of the probability.

A row of probabilities is useful, but often you want one label per sample. You get that label by finding the column with the largest value in each row. This is an argmax operation across columns. The index of the largest entry is the position of the predicted class. The argmax returns a column index, not a class name, so the index still has to be translated before you have a label.

Here the hat over k marks the predicted column for sample i. The expression says: among all columns k, pick the one whose probability p_{ik} is largest. The predicted column index is then used to look up a class name in classes_.

The label dog in the diagram comes from a made-up set of class names, cat, dog, and fish, used only to illustrate the lookup. The row values are the ones from the source. The procedure is what matters: find the largest column, take its index, and translate that index through classes_.

In the source's run, the argmax results agree exactly with the output of predict. That agreement is what you should expect, because predict chooses the class with the highest probability. If your argmax and predict ever disagree on a trained model, that is a signal to check which class labels you are mapping to, rather than assuming the probability logic is wrong.

A column index is only meaningful if you know which class each column stands for. A classifier stores the class labels it learned during fitting in an attribute called classes_. The position of each label in classes_ corresponds to the position of the matching column in the output. So classes_[0] names the class in column 0, classes_[1] names the class in column 1, and so on.

The mapping can look obvious when the labels are plain integers that start at 0 and run consecutively, because the position and the label seem to be the same number. That is exactly why the step is easy to skip. The source points out that classes_ must be used to map positions to names, especially for string labels or for integer labels that are not consecutive. Once the labels are strings like cat, dog, and fish, a position of 1 tells you nothing until you look it up.

Many classifiers also provide a second method, decision_function, which returns decision values instead of probabilities. The two methods are closely related in shape. In the multiclass case, both predict_proba and decision_function return an array with shape (n_samples, n_classes), one row per sample and one column per class. The argmax-and-classes_ procedure recovers the predicted class from decision_function in the same way it does from predict_proba, so both need the same care when turning column positions into names.

The binary case is the one exception. For decision_function, a binary classifier returns only one column of values, which corresponds to the positive class, classes_[1]. The source describes this as a historical convenience. It means that the multiclass recipe of taking an argmax across columns does not apply to that single column in the same way. Instead, you read the one column against the positive class and interpret it as the source describes for that case.