Precision and Recall

Imagine a trained model that looks at each sample and produces a score. The score is not yet a decision. Someone must still decide where the cutoff sits: scores above the cutoff are called positive, and scores below it are called negative. That cutoff is called the threshold. The same trained model can therefore behave very differently depending on which threshold you pick.

Precision and recall are the two measurements that describe this behavior, and they usually pull against each other. Moving the threshold to catch more of the real positive samples can let in more wrong positives, and tightening the threshold can do the reverse. This article teaches you to read that trade-off directly from the numbers, one threshold at a time, rather than trusting a single default setting.

The source describes the extreme case that shows why a threshold cannot be chosen carelessly. If everything is classified as positive, recall is 100%, yet the model is useless. A single high recall number on its own says nothing about whether the model is any good.

A classifier that outputs a score must still turn that score into a yes-or-no decision. The cutoff used for that decision is the threshold. Changing the threshold changes the trade-off between precision and recall for the same trained model.

A threshold can also be set to meet a target. The source gives a concrete case: if a business needs to miss fewer than 10% of positive samples, it needs a recall of 90%. The difficulty is that a threshold which reaches that target may leave the model with poor precision. Hitting the target is only half of the job; the other half is checking what it costs in precision.

Because the right threshold is often unknown when a model is first built, practitioners want to see all possible trade-offs at once. The precision-recall curve is the tool for that. Instead of asking for one threshold, it shows the trade-off for every threshold the model's scores allow.

In Python, the function that computes the curve is precision_recall_curve, found in sklearn.metrics. It needs two inputs: the ground-truth labels and the predicted uncertainties. The predicted uncertainties come from either the decision_function method or the predict_proba method of the model. The function returns three arrays: precision values, recall values, and thresholds.

The returned thresholds cover all possible thresholds, meaning all values that appear in the decision function, in sorted order. Those sorted values are what allow the curve to be drawn. This is why every point on the curve can be traced back to one specific threshold.

The source describes the curve as starting at the top-left corner for a very low threshold, where everything is classified as positive. Raising the threshold moves the curve toward higher precision and lower recall. The source also states that the more a model keeps recall high as precision goes up, the better it is.

Now trace a real sweep. Start at threshold 0.9 and lower the threshold step by step to 0.2. Each lower threshold makes the model label more samples as positive. Before reading any output, decide what you expect recall to do under that change, and whether precision should keep rising, falling, or do something else.

Read the table as evidence. Recall never decreases down the column. Precision does not follow a single direction: it starts at 1.000 at threshold 0.9, drops to 0.500 at 0.8, climbs to 0.667 at 0.7 and to 0.750 at 0.6, and then falls to 0.600 at 0.4 and to 0.500 at 0.2. Each row is one point on the curve, and each point is one threshold's trade-off.

A common expectation is that precision always falls as the threshold drops, because more samples get labeled positive and some of them must be wrong. The printed sweep contradicts that expectation in one specific place, and the printed output is the evidence for finding it.

The place to look is the transition from the threshold 0.8 line to the threshold 0.7 line and onward to the threshold 0.6 line. Precision rises from 0.500 to 0.667 to 0.750 across those lines. The recall column is consistent with the curve idea, but the precision column is not monotonic, so a learner who expected a steady decline in precision finds the discrepancy exactly at those rows.

The source describes a shape feature of its example curve that tells you where a change is cheap and where it is expensive. On the left side the curve is relatively flat, which means recall does not drop much while precision is increased up to about 0.5. For precision greater than 0.5, each gain in precision costs a lot of recall.

This is the kind of information the curve reveals that a single number cannot. It is why an operating point should be chosen by looking at the curve rather than by accepting a default threshold.

An operating point is the threshold chosen to meet a business requirement, such as 90% recall, while still keeping reasonable precision. Fixing an operating point helps an organization make performance guarantees to customers or to other groups inside the organization. The threshold is then set to meet that target, and the remaining question is whether the model still has reasonable precision there.

Different classifiers can perform well in different parts of the curve, which means at different operating points. A support vector classifier (SVC) and a random forest trained on the same dataset can have different curve shapes. A random forest has predict_proba but no decision_function, so its curve is built from the probability of class 1, written rf.predict_proba(X_test)[:, 1]. The default threshold for predict_proba in binary classification is 0.5, so that threshold is the one marked on the random forest curve.

The source makes the point with the F1 score. Its printed values are 0.610 for the random forest and 0.656 for the SVC, which would favor the SVC. But the F1 score captures only one point on the precision-recall curve, the one given by the default threshold. Comparing whole curves can show differences that the F1 score would miss, which is why the source says comparing whole curves gives more detail than a single summary number.

Comparing two curves by eye is a manual process. For an automatic comparison, the source describes summarizing the curve as the integral, or area under it, known as the average precision. This summary does not depend on a single threshold or operating point. It is computed with average_precision_score, and the decision_function or predict_proba output must be passed in, not the output of predict, because the calculation considers multiple thresholds.

Under this summary the random forest and the SVC perform similarly, with the random forest slightly ahead, which differs from the earlier F1 comparison. Because average precision is the area under a curve running from 0 to 1, its value always lies between 0 (worst) and 1 (best). The source also states that the average precision of a classifier that assigns decision function values at random equals the fraction of positive samples in the dataset.