Training and Testing Classifiers in scikit-learn

Classifiers can provide uncertainty estimates in addition to the bare predicted label. These are numeric signals that go beyond the label and describe how strongly the model favors each option.

Programming

Two Test Points With the Same Label

Most first work with a classifier asks one question: for a given test point, which class does the model predict? That answer is useful, but it hides something important. Often you also want to know how sure the classifier is that the predicted class is the right one.

Imagine two test points that both come back labeled 'red'. For the first, the classifier is almost completely certain. For the second, it is barely leaning toward red. Both return the same label, yet they should not be treated the same way. The label alone cannot tell you which situation you are in.

Following the Chain From Data to Fitted Model

The source builds its discussion around a small synthetic two-dimensional dataset and a gradient boosting classifier. Read the steps in order before worrying about uncertainty. The data is generated, the class labels are renamed to words so the output is easier to read, the data is split into training and test portions, and the model is fitted on the training part. Only after that does the model exist as an object you can ask questions of.

1
Generate a synthetic two-dimensional dataset with features X and class labels y
2
Rename the numeric class labels to words such as red and blue so the output is readable
3
Split the data into a training portion and a test portion
4
Create a GradientBoostingClassifier and call fit on the training portion only, producing the trained object gbrt
5
Query the trained gbrt with test points using one of its uncertainty methods

The source's code stops after fitting. The point is that gbrt is now a trained classifier, and every method you call in the next steps belongs to that one object. Keep the name gbrt in mind. If an output later looks wrong, the place to look is the step where the wrong object or the wrong input entered this chain, not only the final print statement.

Data flows in one direction. The training portion is the only input to fit, and the test points are the only input to the uncertainty methods.

Why a Predicted Class Is Not Enough

A predicted class is a single decision. It collapses everything the model learned about one input into one word. Two inputs can land on the same side of the boundary while sitting in very different positions relative to it. A label that says red with near-total confidence and a label that says red while barely leaning that way carry the same surface information, but a careful application would act on them differently.

Key point

Classifiers can provide uncertainty estimates in addition to the bare predicted label. These are numeric signals that go beyond the label and describe how strongly the model favors each option.

The reason to expose uncertainty is that the cost of a wrong answer is not always the same. A model used for decisions with real consequences should make its confidence available, because a wrong answer can be cheap in one setting and expensive in another. The source keeps this discussion general and returns to it in more detail later in the course, so this article only introduces the idea.

Two Methods That Expose Uncertainty

scikit-learn offers two methods for obtaining uncertainty estimates from classifiers: decision_function and predict_proba. Their outputs are not the same kind of thing, which is why you should predict what each one returns before you read the result.

GradientBoostingClassifier has both methods, which makes it a convenient model for comparing them. Calling both on the same fitted object with the same test points means any difference you see comes from the method rather than from the data.

Ask which method you called by looking at the shape and range of what comes back: one signed score per point, or one probability per class per point.

Running the Example and Checking the Shapes

The following script follows the source's workflow: generate two-dimensional data, rename the labels, split, fit gbrt, and then call both methods on the same test points. The script is a generated illustration of the steps described in the source; it does not reproduce the source's exact code.

python

import numpy as np
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier

X, y = make_blobs(n_samples=400, centers=2, random_state=0)
y_named = np.where(y == 0, 'red', 'blue')

X_train, X_test, y_train, y_test = train_test_split(X, y_named, random_state=0)

gbrt = GradientBoostingClassifier(random_state=0)
gbrt.fit(X_train, y_train)

print(gbrt.classes_)

scores = gbrt.decision_function(X_test[:5])
print(scores.shape)
print(scores)

probs = gbrt.predict_proba(X_test[:5])
print(probs.shape)
print(probs)

import numpy as np
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier

X, y = make_blobs(n_samples=400, centers=2, random_state=0)
y_named = np.where(y == 0, 'red', 'blue')

X_train, X_test, y_train, y_test = train_test_split(X, y_named, random_state=0)

gbrt = GradientBoostingClassifier(random_state=0)
gbrt.fit(X_train, y_train)

print(gbrt.classes_)

scores = gbrt.decision_function(X_test[:5])
print(scores.shape)
print(scores)

probs = gbrt.predict_proba(X_test[:5])
print(probs.shape)
print(probs)

Notes on the code

make_blobs creates two-dimensional synthetic data with two centers
np.where renames the numeric labels to words
train_test_split keeps the test portion separate from fit
fit is called only on the training portion
Both uncertainty methods are called on the same fitted object gbrt with the same test points

What do you think happens?

Before you run the code, which call returns one value per test point with a sign, and which returns one column per class whose values sum to 1 for each row?

decision_function returns the signed scores; predict_proba returns per-class probabilities
predict_proba returns the signed scores; decision_function returns per-class probabilities

Reveal answer

Answer:

decision_function returns the signed scores; predict_proba returns per-class probabilities

decision_function gives an unbounded signed value per test point, while predict_proba gives one probability per class for each point, and each row sums to 1.

Run the script and check the shapes first. The printed classes_ list shows the order of the columns in predict_proba. The shape of the decision_function output is one-dimensional for this binary problem, while the predict_proba output has two columns. Checking shapes is the quickest way to confirm which method you are looking at.

Moving the Cutoff on a Probability

A predicted label is produced by applying a rule to the model's output. Once you have a per-class probability, you can apply a different cutoff of your own. Raising the cutoff for the red class means fewer points get called red, and each point that crosses the line changes its predicted class.

python

probs = gbrt.predict_proba(X_test)
red_col = list(gbrt.classes_).index('red')
red_prob = probs[:, red_col]

default_pred = gbrt.predict(X_test)
cautious_pred = np.where(red_prob >= 0.8, 'red', 'blue')

print((default_pred == 'red').sum(), 'red under the default rule')
print((cautious_pred == 'red').sum(), 'red under the 0.8 cutoff')

probs = gbrt.predict_proba(X_test)
red_col = list(gbrt.classes_).index('red')
red_prob = probs[:, red_col]

default_pred = gbrt.predict(X_test)
cautious_pred = np.where(red_prob >= 0.8, 'red', 'blue')

print((default_pred == 'red').sum(), 'red under the default rule')
print((cautious_pred == 'red').sum(), 'red under the 0.8 cutoff')

Notes on the code

The column for red is looked up by name, not assumed to be first
predict applies the model's own default decision rule
The 0.8 cutoff is a choice you make, and it can be changed

Points B and C are called red at a cutoff of 0.5 but flip to blue when the cutoff is raised to 0.8. Point A stays red under both cutoffs. The probabilities here are illustrative values.

Read the diagram as a list of test points in order, each with its probability of red. Moving the cutoff does not change any probability. It only changes which points cross the line and therefore which predicted class they receive. That is the mechanism by which the trade-off between kinds of mistakes can be adjusted.

Mistakes That Carry Different Costs

A mistake can go in two directions. A false positive means the model predicts the positive class for an input that does not belong to it. A false negative means the model predicts the other class for an input that does belong to the positive class. The two are not interchangeable: which one is more costly depends on the application, and the same mistake can be cheap in one setting and expensive in another.

Setting	Costly mistake	Why it matters
Routine recommendation	False positive	A wrong suggestion is cheap to ignore
Serious screening decision	False negative	A missed case can carry a high cost
Automated action with no review	Either direction	Confidence determines whether a human should check the case

Illustrative settings. The right balance depends on the real consequences in each application.

Common mistake
Treating the predicted label as if it carried confidence
What it looks like
Two points both labeled red are handled identically even though one is a near tie
Why it fails
The label collapses the model's confidence into one word
Do this instead
Look at the uncertainty output for the points where the decision matters most
Common mistake
Assuming every classifier has decision_function and predict_proba
What it looks like
Calling predict_proba on a model that only exposes decision_function
Why it fails
Most classifiers have at least one method, but not all have both
Do this instead
Check which method the specific classifier provides before writing code that depends on it
Common mistake
Reading decision_function values as probabilities
What it looks like
Saying a score of 2.0 means a 200 percent chance
Why it fails
decision_function values are unbounded signed scores, not probabilities
Do this instead
Read the sign for the side and the magnitude only for relative distance from the boundary
Common mistake
Fitting on the test data
What it looks like
Passing the full dataset to fit and then evaluating on the same points
Why it fails
The test points are then no longer independent of training
Do this instead
Keep the test portion separate and pass only the training portion to fit

Practice Reading the Two Outputs

MEDIUM

Using the gbrt example, predict the output shape of decision_function and of predict_proba for the first five test points before you run the code. Then run the code and check the shapes. Explain in one sentence what the sign of a decision_function value tells you, and why a predict_proba row always sums to one.

Hints

Think about whether a score needs one value or one value per class
The classes_ attribute tells you the column order for predict_proba
A probability row covers every class, so its values must add up to one

Your attempt (not saved)

Key Takeaways

A predicted class is one decision, and it can hide how confident the classifier is about that decision
scikit-learn exposes uncertainty through decision_function, which returns unbounded signed scores, and predict_proba, which returns per-class probabilities that sum to one for each point
Not every classifier has both methods, so check which ones a given model provides and confirm shapes before interpreting outputs
Mistakes can have very different consequences, so the cutoff on a score or probability should reflect the real cost of false positives and false negatives, and test data must stay separate from fit

Related concepts

Evaluating Classifiers Beyond Accuracy

More concepts starting with T