Decision Thresholds and decision_function

Imagine a classifier whose scores are assigned completely at random, with no information about the labels at all. Now imagine a dataset in which 30% of the samples are positive. Before reading on, make a guess: what would the summary number for this meaningless classifier look like?

That baseline is a useful starting point. A single summary number only means something when you know what a useless model would score. This article builds the tool that shows the full picture: the precision-recall curve, which describes every threshold a classifier could use rather than one fixed decision.

A classifier that outputs a score for each sample is turned into a yes-or-no decision by picking a threshold. Samples whose score is at or above the threshold are called positive, and the rest are called negative. Move the threshold and you change how many samples get called positive. That change in turn changes precision and recall, so the two measures trade against each other as the threshold moves. The same classifier can look very different depending on where the threshold sits.

In scikit-learn, the decision_function method produces these continuous scores for models such as an SVM. The default threshold of decision_function is zero. The predict method uses that default, so it commits to one particular trade-off without showing you the alternatives.

The key point is that a single threshold gives you a single pair of precision and recall. The curve is what you get when you repeat this pipeline for every threshold the classifier could use, rather than stopping at the default.

The function that computes the curve lives in the sklearn.metrics module. It takes two inputs: the ground-truth labels and a continuous certainty score for the positive class. That score can come from decision_function or from predict_proba. Passing scores rather than final predicted labels is what lets the function explore many thresholds.

The call returns three sequences that line up by index. The thresholds are every distinct value that appears in the decision function, in sorted order. The precision and recall entries at position i are the values you would get by using the threshold at position i. Plotting precision on one axis and recall on the other turns those pairs into a curve.

Each point on the curve belongs to exactly one threshold. A marked point on the curve corresponds to threshold zero, the default of decision_function. Every other point is a trade-off you could have chosen instead. Seeing the default among all the alternatives is the main reason to draw the whole curve.

Once the curve is drawn, you can read trade-offs directly from it without rerunning the model at a new threshold. The source example uses an SVC trained with gamma set to 0.05. On that curve, a recall of 0.4 can be reached at a precision of about 0.75. You read the pair straight off the plot.

The shape of the curve also shows where precision is cheap or expensive. In the same example, precision can reach about 0.5 while recall stays very high, so the left part of the curve is relatively flat. Past a precision of about 0.5, each further gain in precision costs a large amount of recall. Those steep stretches are where asking for more precision is a costly choice.

The curve also describes the overall pattern. Good curves stay near the upper-right corner, where precision and recall are both high at the same threshold. The source describes the curve as beginning at the top-left corner, where the threshold is so low that every sample is called positive. As the threshold rises, the curve moves toward higher precision and lower recall. At very high thresholds, the few samples called positive are almost all correct, so precision is high while recall has dropped. A better model keeps recall high for as long as possible while precision climbs.

A specific threshold or requirement placed on a classifier is called its operating point. A requirement such as 90% recall is one example. In many projects the right threshold comes from a business requirement, such as missing no more than 10% of the positive cases. Choosing an operating point is useful in business settings because it lets a team make a performance promise to customers or to other groups inside the organization.

Once a target like that is fixed, a threshold can usually be found to meet it. The harder question is whether the model can still keep precision reasonable at that threshold. A model that labels every sample as positive reaches 100% recall, but it is useless. Judging a model well therefore means looking at how precision and recall move together, not at either one alone.

An operating point is still just one point on the curve. The curve is what tells you whether that point is realistic. If the curve drops sharply at 90% recall, the operating point is expensive to hold, and the team may need to revisit the requirement or improve the model.

Different classifiers can work well in different parts of the curve, which is why comparing their curves is more informative than comparing a single number. The source example compares an SVM with a random forest trained on the same dataset. The random forest has no decision_function, only predict_proba, so the probability of class 1 is passed to the precision-recall function as the certainty score.

Ranking the two by F1 score alone says the SVM is clearly ahead. That ranking is misleading as a complete picture. The F1 score captures only one point on the curve, the one set by the default threshold. Whatever the curves do at other thresholds is invisible to it, and so is the difference at the extremes where the random forest wins.

Comparing curves by eye gives detail, but it is a manual process. For automatic model comparison, you want a single number that summarizes the curve without committing to one threshold or operating point. One such summary is the area under the precision-recall curve, which is called average precision. The average_precision_score function computes it.

Notice the argument. Average precision needs every threshold to be considered, so it takes the continuous scores from decision_function or predict_proba. Passing the output of predict would collapse those scores into hard labels and remove the threshold information the summary depends on.

Average precision summarizes the curve as its area and ranges from 0 to 1. Since it is the area under a curve that runs from 0 to 1, its value always falls between 0, the worst case, and 1, the best case. A random scorer is expected to reach the positive fraction, which is the baseline you saw in the opening question.

In the SVM and random forest example, averaged over all thresholds, the two classifiers come out nearly tied, with the random forest slightly ahead. This reverses the F1 ranking from the previous section. That is exactly why the summary and the single-threshold score should be read together.