Understanding Classifier Uncertainty

A classifier can provide uncertainty estimates in addition to its predicted class, which matters when different mistakes lead to very different outcomes.

Programming

Why One Label Hides Part of the Story

Most first work with a classifier asks one question: for a given test point, which class does the model predict? That answer is useful, but it hides something important. Often you also want to know how sure the classifier is that the predicted class is the right one.

Consider two test points that both come back labeled red. One is a point the classifier calls red with near-total confidence. The other is a point where the classifier is barely leaning toward red. Both return the same label, yet the source is clear that they should be treated differently. A bare label cannot tell you which situation you are in.

scikit-learn gives you a way to get at this extra information. A classifier can provide uncertainty estimates alongside its predictions: a numeric signal that goes beyond the bare label. The signal matters because the cost of a wrong answer is not always the same, and that idea is the thread running through the rest of this article.

Two test points receive the same predicted class, but one lies far from the decision boundary and the other lies close to it. The label alone does not separate them.

Tracing the Fitted Model

The source builds its discussion around a small synthetic two-dimensional dataset and a gradient boosting classifier. Before looking at any uncertainty method, follow the steps that produce a trained model. The data is generated, the class labels are renamed to words so the output is easier to read, the data is split into training and test portions, and the model is fitted on the training part.

1
Generate a synthetic two-dimensional dataset with class labels
2
Rename the numeric class labels to words such as red and blue
3
Split the data into a training portion and a test portion
4
Create a GradientBoostingClassifier and fit it on the training portion, naming the fitted object gbrt
5
Query gbrt with test points using one of its methods

python

from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
import numpy as np

X, y = make_blobs(n_samples=200, centers=2, random_state=0)
names = np.where(y == 0, "red", "blue")
X_train, X_test, y_train, y_test = train_test_split(X, names, random_state=0)

gbrt = GradientBoostingClassifier(random_state=0)
gbrt.fit(X_train, y_train)

from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
import numpy as np

X, y = make_blobs(n_samples=200, centers=2, random_state=0)
names = np.where(y == 0, "red", "blue")
X_train, X_test, y_train, y_test = train_test_split(X, names, random_state=0)

gbrt = GradientBoostingClassifier(random_state=0)
gbrt.fit(X_train, y_train)

Notes on the code

make_blobs generates a small synthetic two-dimensional dataset
np.where renames the numeric labels to words
train_test_split creates separate training and test portions
gbrt is the fitted classifier that every later call uses

Notice that nothing in this code yet asks for uncertainty. The code stops after fitting. That is the point: gbrt is now a trained classifier, and the next steps examine two methods that this one object exposes. Keep the name gbrt in mind, because each method you call is a method of that same object.

Best practice

When a result looks wrong, trace it back along the chain: data, renamed labels, training split, fitted model, then the query. The useful place to look is the step where the wrong object or the wrong input entered the chain, not the final print statement alone.

Two Methods, Two Kinds of Output

scikit-learn offers two methods for getting uncertainty estimates from a classifier: decision_function and predict_proba. The source says GradientBoostingClassifier has both, which makes it a convenient model for comparing them. Both methods are called on the same fitted object with the same test points, so any difference you see comes from the method rather than from the data.

Both methods are called on the same fitted model with the same test points. One returns a raw score per point, the other returns values that behave like per-class likelihoods.

The two methods are not described as producing the same kind of result, and the difference is worth predicting before you check. Ask yourself what a single number per test point would look like, compared with a set of values that behave like likelihoods for each class.

Following One Sample Across the Decision Boundary

Now trace what happens to a single sample when you read a decision_function score. In a two-class setting, the score sits on a line. The zero point is the boundary between the two sides, so a sample's position relative to zero determines its predicted class. The farther a score is from zero, the farther the sample is from the boundary.

Each sample has a decision_function score. Its position relative to the zero boundary determines the predicted class, and its distance from zero indicates how far it lies from the boundary. The scores shown are illustrative.

Sample B and Sample C are both close to the boundary, so a small change in the data could move them to the other side. Sample A and Sample D are far from it. The predicted label alone would report Sample B and Sample A as the same kind of answer, even though their positions tell very different stories about how firm the decision is.

Mistakes Beginners Make with Uncertainty Outputs

Common mistake
Assuming every classifier has both decision_function and predict_proba
What it looks like
Writing code that calls predict_proba on a model that only offers decision_function
Why it fails
The source says most classifiers have at least one of the two methods and many have both, but not all have both.
Do this instead
Check which method a given classifier provides before you depend on it.
Common mistake
Treating a decision_function score as a probability
What it looks like
Reading a score of 1.8 as an 180 percent chance of a class
Why it fails
A decision_function score is a position relative to the decision, not a likelihood, and the two methods are not described as producing the same kind of result.
Do this instead
Read decision_function scores as signed positions, and read predict_proba values as per-class likelihood-like numbers.
Common mistake
Comparing methods on different test points
What it looks like
Calling decision_function on one set of points and predict_proba on another, then comparing the outputs
Why it fails
The comparison is only meaningful when both methods are called on the same fitted object with the same test points.
Do this instead
Use one fitted model and one set of test points for every comparison.
Common mistake
Judging a prediction only by its label when the surrounding uncertainty would change the action taken

Why Different Mistakes Cost Different Amounts

The reason to bother with uncertainty estimates is that a wrong answer does not always carry the same consequence. A model used for decisions with real consequences should expose how confident it is, because a wrong answer can be cheap in one setting and expensive in another. The source keeps this discussion general and says it will return to the topic in more detail in Chapter 6, so this article only introduces the idea.

A false positive and a false negative are both errors, but in a given application they can lead to very different real-world outcomes. This illustrative pair shows how the same model error can carry different weight.

Predict, Then Check

What do you think happens?

In the example above, gbrt.decision_function(X_test[:5]) and gbrt.predict_proba(X_test[:5]) are both called. What shape do you expect each output to have?

decision_function returns one value per test point, and predict_proba returns one value per class for each test point
Both return one value per test point with the same shape
decision_function returns a probability for each class, and predict_proba returns a signed score

Reveal answer

Answer:

decision_function returns one value per test point in this two-class case, and predict_proba returns one row per test point with one column per class.

The score from decision_function is a signed position relative to the decision, so one number per point is enough. predict_proba gives a likelihood-like value for each class, so each point gets a row with one entry per class.

MEDIUM

Without running any code, describe in your own words how you would decide whether to trust a red prediction for a test point. Say which output you would read, what it tells you about distance from the decision boundary, and why that might matter more in one application than in another.

Hints

Think about the difference between a score that is far from zero and one that is close to zero
Consider what happens when a wrong red prediction is costly compared with a wrong blue prediction

Your attempt (not saved)

Key Takeaways

A predicted class alone can hide how sure the classifier is, and two points with the same label may deserve different treatment.
scikit-learn exposes uncertainty through decision_function and predict_proba, but not every classifier provides both, so check which methods a given model offers.
decision_function returns a signed score per test point, while predict_proba returns a likelihood-like value for each class.
Different mistakes can carry very different consequences, so the uncertainty signal helps decide which predictions need closer attention.

More concepts starting with U