Choosing Decision Thresholds
Scores above the threshold are called positive, and scores below it are called negative.
One Set of Predictions, Many Stories
Many classifiers do not answer yes or no directly. They produce a score for each sample, and a threshold converts that score into a decision. Scores above the threshold are called positive, and scores below it are called negative. Change the threshold and the decisions change. That means a single set of predictions tells you only one story about the model, and the story depends on where the cut was placed.
What do you think happens?
Four samples have scores 0.9, 0.8, 0.7, and 0.1. If the threshold is 0.85, how many samples are predicted positive?
Reveal answer
One. Only the score 0.9 is above 0.85.
Scores strictly above the threshold are called positive. The other three scores fall at or below 0.85, so they are predicted negative.
Scores Become Decisions at a Threshold
A threshold is a cut point on a continuous score. Every sample whose score lies above the cut is labeled positive, and every sample below it is labeled negative. The model itself does not change when you move the cut. Only the mapping from scores to decisions changes, so the same scores can produce very different confusion tables.
This is why a threshold is a decision, not a property of the scores. Lowering the threshold can catch more real positives, but it can also let more negatives through as false alarms. Raising it does the opposite. Any evaluation that reports one set of predictions has quietly picked one threshold, and it should be read with that in mind.
Sweeping Every Threshold at Once
The receiver operating characteristic curve, usually called the ROC curve, avoids picking one threshold. It looks at all possible thresholds for a classifier together. For each threshold it computes two quantities: the false positive rate on the horizontal axis and the true positive rate on the vertical axis. The true positive rate is the same quantity as recall, so the vertical axis is a measure you already know from precision-recall work.
Each threshold gives one point, and moving the threshold from one extreme to the other traces the curve from bottom left to top right. The ideal curve sits near the top left corner, where the true positive rate is high and the false positive rate is low. In that corner the classifier finds most of the positives while producing few false alarms.
The shape also tells you how good the classifier is. A curve that hugs the diagonal from bottom left to top right means the true and false positive rates rise together, which is close to guessing. The further the curve bends toward the top left, the better the classifier separates the two classes across thresholds.
Counting the Four Cells
Both axes of the ROC curve are built from the four counts of a binary confusion table: true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN). The true positive rate is the fraction of all actual positive samples that the classifier catches. The false positive rate is the fraction of all actual negative samples that were wrongly called positive.
| Rate | Formula | Numerator counts | Denominator counts | What it measures |
|---|---|---|---|---|
| TPR (recall) | $\frac{TP}{TP + FN}$ | True positives: positives that were caught | All actual positives (TP + FN) | Share of real positives the model finds, so misses are the error being tracked |
| FPR | $\frac{FP}{FP + TN}$ | False positives: negatives wrongly called positive | All actual negatives (FP + TN) | Share of real negatives that became false alarms, so the cost of false positives relative to the negative class |
The numerator and denominator of each rate come from different true classes, which is why the two rates measure different kinds of error.
Notice that each rate divides by the size of one true class. That keeps the two rates comparable even when the classes are very unequal in size. FPR is the piece that reveals false alarms: a model can have a high TPR while still producing many false alarms, and the FPR is what exposes that.
Worked Example: Four Samples by Hand
Computing TPR and FPR at threshold 0.75
Four samples have true labels 1, 0, 1, 0 and classifier scores 0.9, 0.8, 0.7, and 0.1. Find TPR and FPR when the threshold is 0.75, where scores strictly above 0.75 are predicted positive.
- 1Find the predicted positives
Scores above 0.75 are 0.9 and 0.8. The scores 0.7 and 0.1 are predicted negative.
- 2Match them to true labels
The sample scoring 0.9 has true label 1, so it is a TP. The sample scoring 0.8 has true label 0, so it is an FP.
- 3Count the remaining cells
The sample scoring 0.7 has true label 1 and is predicted negative, so it is an FN. The sample scoring 0.1 has true label 0 and is predicted negative, so it is a TN. That gives TP = 1, FN = 1, FP = 1, TN = 1.
- 4Compute TPR
TPR = TP / (TP + FN) = 1 / (1 + 1) = 0.5.
- 5Compute FPR
FPR = FP / (FP + TN) = 1 / (1 + 1) = 0.5.
At threshold 0.75 the point on the ROC curve is (FPR, TPR) = (0.5, 0.5).
The same counting procedure works at every threshold. Lowering the threshold to 0.5 brings the samples scoring 0.9, 0.8, and 0.7 into the positive group. That yields TP = 2, FN = 0, FP = 1, TN = 1, so TPR = 1 and FPR = 0.5, which is the point (0.5, 1) on the curve.
Calling roc_curve and roc_auc_score
In practice you do not count cells by hand. The roc_curve function returns three arrays: the false positive rate at each threshold, the true positive rate at each threshold, and the thresholds themselves. Each index i describes one operating point of the classifier, and plotting the FPR array against the TPR array draws the curve. The roc_auc_score function takes the true labels and the continuous scores and returns the area under that same curve.
from sklearn.metrics import roc_curve, roc_auc_score
y_true = [1, 0, 1, 0]
scores = [0.9, 0.8, 0.7, 0.1]
fpr, tpr, thresholds = roc_curve(y_true, scores)
print('fpr:', fpr)
print('tpr:', tpr)
auc = roc_auc_score(y_true, scores)
print('AUC:', auc)- 1Line 1 imports both functions from sklearn.metrics.
- 2Lines 3 and 4 define the four true labels and the four scores from the worked example.
- 3
roc_curvepairs each candidate threshold with its FPR and TPR, which is why the arrays are the same length as the thresholds. - 4The printed FPR and TPR arrays match the hand-computed points (0, 0), (0, 0.5), (0.5, 0.5), (0.5, 1), and (1, 1).
- 5
roc_auc_scorereturns 0.75, the area under those points, which matches the ranking count in the next section.
y_trueholds the true labels, and scores holds the classifier's continuous output for each sample.roc_curvereturns the FPR and TPR at each operating point, plus the thresholds that produced them.roc_auc_scoreuses the scores directly and does not need you to pick a threshold.
fpr: [0. 0. 0.5 0.5 1. ]
tpr: [0. 0.5 0.5 1. 1. ]
AUC: 0.75AUC as a Ranking Probability
Just as the precision-recall curve can be summarized with one number, the ROC curve can be summarized with the area under it, almost always called the AUC. Because the curve runs from bottom left to top right, the area is a single value that describes performance across all thresholds together.
- 1s+ is the score of a randomly chosen positive sample.
- 2s- is the score of a randomly chosen negative sample.
- 3The probability is taken over all such random pairs, so AUC answers: how often does a random positive receive a higher score than a random negative?
- 4Because this reads as a ranking statement, AUC does not depend on any single threshold.
You can check the ranking view on the four-sample data. The positives are the samples scoring 0.9 and 0.7, and the negatives are the samples scoring 0.8 and 0.1. There are four positive-negative pairs. The pair 0.9 against 0.8 is correctly ranked, 0.9 against 0.1 is correctly ranked, 0.7 against 0.1 is correctly ranked, but 0.7 against 0.8 is not. Three of four pairs are correct, so the AUC is 0.75, the same value roc_auc_score returned.
On this scale, 0.5 corresponds to random performance and 1 corresponds to perfect separation, where every positive outranks every negative.
Comparing Models on the Same Test Data
The source compares two models on the same test data: a random forest and an SVC. The AUC for each is computed from the quantity that model naturally produces. The random forest supplies predicted probabilities for the positive class, and the SVC supplies scores through its decision function. Because roc_auc_score only needs a continuous score that ranks samples, either output works.
Random forest AUC: 0.937
SVC AUC: 0.916- rf and svc are assumed to be trained models, and X_test and
y_testare assumed to be the same test split. predict_proba(...)[:, 1] selects the probability of the positive class.decision_functionreturns signed scores, which are also acceptable for ranking.
On this data the random forest scores 0.937 and the SVC scores 0.916, so the random forest performs noticeably better by this measure. Note that these values come from the source's data and models, which the snippet assumes already exist.
Why Accuracy Hides Differences on Imbalanced Data
The source then turns to a harder case: classifying all nines in the digits dataset against all other digits. The nines are a small share of the samples, so the problem is imbalanced. An SVM is trained with three different settings of the kernel bandwidth, gamma, and for each setting the source records accuracy and AUC and draws the ROC curve.
| Gamma setting | Accuracy | AUC | What the ranking shows |
|---|---|---|---|
| 1.0 | 90% | At chance level | The decision function is no better than random at ranking positives above negatives |
| 0.01 | 90% | 1.0 | Every positive sample is ranked above every negative sample |
Accuracy is identical across these settings, while the AUC separates them. Only the two anchoring settings named in the source are shown.
The accuracy is 90% for all three gamma settings. Accuracy alone would suggest the three models are equivalent, and the source notes that this could be chance-level performance or might not be. The AUC and the curves separate them clearly. In the source's words, with the right threshold the gamma=0.01 model can classify the data perfectly, and adjusting the threshold would yield strong predictions. Accuracy alone would never have revealed that.
Mistakes With Thresholds and AUC
Common mistake Treating the default threshold as the best threshold
What it looks likeReporting precision and recall at the default cut and assuming no other cut would work better.
Why it failsThe ROC curve shows that other operating points can reach higher recall with only a small rise in false positive rate.
Do this insteadInspect the curve and choose an operating point on a validation set that matches the trade-off you need.
Common mistake Choosing the threshold on the test set
What it looks likeScanning the test-set ROC curve for the best point and then reporting the score at that point.
Why it failsThe threshold was tuned to the test data, so the reported performance is optimistic.
Do this insteadSelect the threshold on a separate validation set, then evaluate once on the test set.
Common mistake Judging models by accuracy alone on imbalanced data
What it looks likeThree SVMs all show 90% accuracy on the nines problem, so they are declared equivalent.
Why it failsAccuracy hides the ranking differences. One model had AUC 1.0 and another was at chance level.
Do this insteadCompare models with the ROC curve and AUC, and then tune the threshold for the chosen model.
Common mistake Reading AUC as a threshold
What it looks likeReading an AUC of 0.95 as meaning the model should cut scores at 0.5.
Why it failsAUC summarizes ranking across all thresholds and does not specify any one of them.
Do this insteadAfter confirming the AUC, pick a threshold on a validation set using the trade-off you need.
Common mistake Dividing FPR by the wrong count
What it looks likeComputing FPR as FP divided by the number of predicted positives.
Why it failsFPR divides false positives by the number of actual negatives, which is FP + TN, so the rate measures cost relative to the negative class.
Do this insteadUse FPR = FP / (FP + TN) and TPR = TP / (TP + FN).
Practice: Moving the Cut Point
Use the same four samples with labels 1, 0, 1, 0 and scores 0.9, 0.8, 0.7, and 0.1. Set the threshold to 0.85. Count TP, FP, FN, and TN, then compute TPR and FPR. Where does this point sit on the ROC curve relative to the point at threshold 0.75?
Hints
- Only scores strictly above the threshold are predicted positive.
- At 0.85, only the score 0.9 is above the cut. Check that sample's true label to decide whether it is a TP or an FP.
- Use $\frac{TP}{TP + FN}$ for TPR and $\frac{FP}{FP + TN}$ for FPR with the counts you found.
Summary of the Threshold Story
A threshold turns a continuous score into a decision, and changing it changes the predictions. The ROC curve sweeps all thresholds at once, plotting FPR against TPR with one point per threshold. AUC condenses the curve into a ranking measure equal to the probability that a random positive outranks a random negative. Use the curve and AUC to compare models, especially on imbalanced data, and then choose the final threshold on a validation set.
At a glance
- A classifier's score becomes a decision only after a threshold is applied, so one set of predictions reflects only one threshold.
- Each threshold yields one (FPR, TPR) point, and sweeping thresholds traces the ROC curve from the bottom left toward the top right.
- TPR = TP / (TP + FN) is recall, and FPR = FP / (FP + TN) measures false alarms relative to the number of actual negatives.
- AUC equals the probability that a random positive ranks above a random negative, with 0.5 for random and 1 for perfect separation, and it can reveal differences that accuracy hides on imbalanced data.
- Choose the final threshold on a validation set, because AUC does not specify a threshold and tuning on the test set makes reported performance optimistic.