Precision-Recall Curves

Many classifiers do not answer yes or no directly. They output a score for each sample, and a threshold turns that score into a decision: scores above the threshold are called positive, and scores below it are called negative. Change the threshold and the decisions change, so a single set of predictions tells you only one story about the model.

The right threshold is often unknown when a model is first built. The threshold can also be set to meet a target. If a business needs to miss fewer than 10% of positive samples, it needs a recall of 90%. The difficulty is that a threshold which reaches that target may leave the model with poor precision. The source makes the extreme case concrete: if everything is classified as positive, recall is 100%, but the model is useless.

A precision-recall curve plots precision against recall for every possible threshold, so each point on it is one threshold's trade-off. In this article you will first see how precision and recall move as the threshold changes, then learn how the curve is computed, and finally how to choose an operating point and compare two classifiers.

The verified printed output lists one threshold per line together with the precision and recall obtained at that threshold. The table below restates those values so you can read them side by side. Read them one row at a time, starting from the highest threshold.

Read the sweep as a walk from the highest threshold toward the lowest. Recall starts at 0.333 at thresholds 0.9 and 0.8, rises to 0.667 at 0.7, reaches 1.000 at 0.6, and stays at 1.000 at 0.4 and 0.2. It never decreases. Precision starts at 1.000 at threshold 0.9, falls to 0.500 at 0.8, rises to 0.667 at 0.7 and to 0.750 at 0.6, and then falls to 0.600 at 0.4 and 0.500 at 0.2. Precision therefore moves in both directions while recall only climbs or holds steady.

The function that computes the curve is precision_recall_curve, found in sklearn.metrics. It needs two inputs: the ground-truth labels and the predicted uncertainties, which come from either the decision_function method or the predict_proba method. It returns three arrays: precision values, recall values, and thresholds. The thresholds cover all possible values that appear in the decision function, in sorted order, and those sorted values are what allow the curve to be drawn.

Each row of the sweep is one point on the curve. The points are ordered by threshold, so moving the threshold from high to low walks along the curve from the low-recall end toward the high-recall end. Reading the curve correctly depends on its direction: the closer a curve stays to the upper-right corner, the better the classifier, because a point in the upper right means high precision and high recall for the same threshold. The source describes the curve as starting at the top-left for a very low threshold, where everything is classified as positive, and raising the threshold moves the curve toward higher precision and lower recall. The more a model keeps recall high as precision goes up, the better it is.

A common expectation is that precision always falls as the threshold drops, because more samples get labeled positive and some of them are wrong. The printed sweep does not behave that way everywhere. Between threshold 0.8 and threshold 0.6, precision rises from 0.500 to 0.750 while the threshold is being lowered. The place to look is the transition from the threshold=0.8 line to the threshold=0.7 line and on to the threshold=0.6 line.

The receiver operating characteristics curve, usually called the ROC curve, looks at all possible thresholds for a classifier at once. Where the precision-recall curve reports precision and recall, the ROC curve reports the false positive rate on the horizontal axis and the true positive rate on the vertical axis. The true positive rate is just another name for recall, so the vertical axis is a measure you already know.

Both axes are built from the four counts of a binary confusion table: true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN). Each axis divides by the size of one true class, which keeps the two rates comparable even when the classes are very unequal in size. The true positive rate divides by the actual positives, and the false positive rate divides by the actual negatives.

The false positive rate measures how much cost the false positives impose relative to the size of the negative class. A model can have a high TPR while still producing many false alarms, and the FPR is what exposes that. Each threshold gives one ROC point, and moving the threshold from one extreme to the other traces the curve from the bottom left to the top right. The ideal curve sits near the top left corner, where the TPR is high and the FPR is low.

Start with a case small enough to check by hand. Four samples have true labels 1, 0, 1, 0, and a classifier gives them the scores 0.9, 0.8, 0.7, and 0.1. The two samples labeled 1 are the positives, and the two labeled 0 are the negatives.

To check the 0.75 by hand, pair every positive with every negative and ask whether the positive received the higher score. The positives are 0.9 and 0.7; the negatives are 0.8 and 0.1. The pair 0.9 against 0.8 is correct, 0.9 against 0.1 is correct, 0.7 against 0.8 is wrong, and 0.7 against 0.1 is correct. Three of the four pairs are ranked correctly, giving 0.75.

AUC summarizes the ROC curve in one number. It equals the probability that a random positive sample ranks above a random negative sample, which is the ranking statement you just checked by hand. Random performance gives 0.5, and perfect separation gives 1.

In this formula s+ is the score of a randomly chosen positive sample and s- is the score of a randomly chosen negative sample. Because AUC is defined through ranking rather than through any one threshold, it describes performance across all thresholds together. The ROC curve itself is produced by roc_curve, which returns the false positive rate, the true positive rate, and the thresholds, one entry per operating point.

The source's harder case is classifying all nines in the digits dataset against all other digits. The nines are a small share of the samples, so the problem is imbalanced. An SVM is trained with three gamma settings, which control the kernel bandwidth, and accuracy and AUC are recorded for each.

Accuracy is 90% for all three gamma settings, so accuracy alone would suggest the three models are equivalent. The source notes that this could be chance-level performance or might not be. The AUC values separate them clearly. With gamma=1.0 the AUC is at chance level, so the decision function is no better than random. With gamma=0.01 the AUC is 1.0. In the source's words, with the right threshold this model can classify the data perfectly. Accuracy alone would never have revealed that.

An operating point is the threshold chosen to meet a requirement, such as 90% recall, while still keeping reasonable precision. Fixing an operating point is useful in business settings because it lets an organization make performance guarantees to customers or to other groups. The threshold is then set to meet that target, and the remaining question is whether precision at that threshold is still acceptable.

The curve shows the shape of this trade-off. On the left side of the source's example curve it is relatively flat, which means recall does not drop much while precision is increased up to about 0.5. For precision above 0.5, each gain in precision costs a lot of recall. This is why an operating point should be chosen by looking at the curve rather than by defaulting to a threshold.

Different classifiers can perform well in different parts of the curve, which means at different operating points. A support vector classifier (SVC) and a random forest trained on the same dataset can have different curve shapes. A random forest supplies probabilities through predict_proba and has no decision_function, so its curve is built from the probability of class 1, written rf.predict_proba(X_test)[:, 1]. The default threshold for predict_proba in binary classification is 0.5, and that is the threshold marked on the random forest curve.

The source reports that the random forest performs better at the extremes, for very high recall or very high precision requirements. Around the middle, at approximately precision 0.7, the SVM performs better. This is why comparing whole curves gives more detail than a single summary number taken at one threshold.

The F1 score illustrates the risk of a single number. The source's printed values are 0.610 for the random forest and 0.656 for the SVC, which would favor the SVC. But the F1 score captures only one point on the precision-recall curve, the one given by the default threshold. Comparing whole curves can show differences that the F1 score would miss.

Average precision summarizes the area under the curve in a single value. Under this summary the random forest and the SVC perform similarly, with the random forest slightly ahead, which differs from the earlier F1 comparison. Because the value is the area under a curve running from 0 to 1, it always lies between 0 (worst) and 1 (best). The source also states that the average precision of a classifier that assigns decision function values at random equals the fraction of positive samples in the dataset.