Evaluating Classifiers with Thresholds

Many classifiers offer a method called predict_proba.

Programming

A Model That Answers With Numbers

Many classifiers offer a method called predict_proba. Instead of handing back one label per sample, it returns a number for each class that says how likely that class is, according to the model. That shift from labels to likelihoods is the foundation for evaluating a classifier with thresholds, because a threshold can only be applied once you have numbers to compare against it.

What do you think happens?

A binary classifier is given 5 samples and asked for predict_proba. What shape do you expect the returned array to have?

(5,)
(5, 2)
(2, 5)
(5, 5)

Reveal answer

Answer:

(5, 2)

The first number is the number of samples, so each sample gets its own row. The second number is 2 because a binary problem has exactly two classes, and the method gives one probability for each of them.

Reading the Shape of the Output

The shape pattern for binary classification is always (n_samples, 2). Read it as a grid. Each row belongs to one sample, so the number of rows equals the number of samples you passed in. Each column belongs to one class, so there are always two columns. If your data had three classes, the same idea would suggest a different second dimension, but this article stays with the two-class case.

Each of the 5 rows holds one sample's two probabilities, so the full array has shape (5, 2). The illustrative values shown are made up for teaching; each row still sums to 1.

python

probs = model.predict_proba(X_test[:6])
print(probs.shape)

probs = model.predict_proba(X_test[:6])
print(probs.shape)

Output (expected)

(6, 2)

Notes on the code

Slicing X_test[:6] passes only the first six test rows, which keeps the output short enough to read.
The shape (6, 2) means six rows, one per sample, with two columns, one per class.

Each Row Is a Pair of Probabilities

Each row is a pair of numbers. The first entry is the estimated probability of the first class, and the second entry is the estimated probability of the second class. The position of a value is what gives it meaning, so the column you read determines which class you are talking about.

One sample's row links to two values: column 0 holds the probability of the first class and column 1 holds the probability of the second class. The values shown are illustrative.

Because each value is a probability, every entry lies between 0 and 1. Because the two classes cover all the possibilities for a sample, the two entries in a row always add up to 1. So if the first entry is 0.82, the second must be 0.18. This is why a single row carries the complete picture for one sample: knowing one column tells you the other.

Checking That a Row Is Valid

A quick check catches most bad outputs. For a row with entries p_0 and p_1, the output is valid when each entry satisfies $0 \leq p \leq 1$ and the two entries satisfy $p_0 + p_1 = 1$. Both conditions are cheap to verify by eye, so make them your first reflex whenever a probability looks odd.

Worked example

Testing one row

A row of predict_proba output reads 0.30 and 0.80. Is this a valid row of probabilities?

1
Check the range
Both 0.30 and 0.80 lie between 0 and 1, so the range condition holds.
2
Check the sum
Adding the two entries gives 0.30 + 0.80 = 1.10, which is not 1.
3
Decide
The sum condition fails, so this row is not a valid probability row. The output should be treated as a sign that something upstream went wrong.

Result

Invalid: the entries are in range but do not sum to 1.

Tracing Where a Wrong Value Starts

Suppose a program is supposed to report how confident the model is about the second class, but the printed number looks wrong. Do not guess at the cause. Work through the program one line at a time and check the result after each step. The first line where the value fails to match your expectation is the place to focus.

After the call to predict_proba, print the shape. It should be (number of samples, 2).
Print one row, such as the first, and confirm its two entries lie between 0 and 1 and sum to 1.
Identify the column the code reads and confirm it matches the class you intended.
The first step that disagrees with your expectation is where the output first diverged.

Mistakes Beginners Make with Probability Output

Common mistake
Expecting one label per sample
What it looks like
Printing predict_proba output and assuming it looks like a list of class names.
Why it fails
predict_proba returns a number for each class for each sample, so the result is a two-dimensional array.
Do this instead
Expect the shape (n_samples, 2) and read each row as a pair of probabilities.
Common mistake
Reading the wrong column
What it looks like
Reporting the first column as the confidence in the second class.
Why it fails
Each column belongs to a specific class, and the first column is the first class's probability.
Do this instead
Confirm which class each column corresponds to before you report or compare values.
Common mistake
Skipping the validity check
What it looks like
Accepting a row of 0.30 and 0.80 without noticing that the entries sum to 1.10.
Why it fails
A valid row has entries between 0 and 1 that add up to 1, so a failed sum signals a problem.
Do this instead
Verify the range and the sum for a row whenever a value looks unexpected.
Common mistake
Assuming the second dimension is always 2 for every problem: it is 2 for the two-class case this article covers, and a multi-class problem would need a different reading.

Practice Reading a Probability Row

EASY

A model returns a row of [0.64, 0.36] for one sample. Which class does the first entry describe, is the row valid, and what shape would the full output have for 4 samples?

Hints

The first entry is the estimated probability of the first class.
Check that each value is between 0 and 1 and that the two values add up to 1.
The number of rows equals the number of samples, and the number of columns is 2 for binary classification.

Your attempt (not saved)

MEDIUM

Given a predict_proba output with shape (6, 2), explain which dimension tells you the number of samples and which tells you the number of classes, then name the first step you would check if the second column looked wrong.