Evaluating Classifiers with Accuracy and Recall

Imagine you have trained a classifier and want to know how good it is. The obvious first step is to look at its predictions and count how many it got right. But many classifiers do not hand you a yes or no answer directly. They output a score for each sample, and a threshold then turns that score into a decision. Change the threshold and the decisions change, so a single set of predictions tells you only one story about the model. This article introduces the ROC curve, which looks at all possible thresholds for a classifier at once, and the AUC, a single number that summarizes that curve.

A score-producing classifier assigns each sample a number. The threshold is the cutoff you pick. Scores above the threshold are called positive, and scores below it are called negative. If you choose a high threshold, few samples pass and the classifier makes few positive calls. If you lower the threshold, more samples pass and the classifier makes more positive calls. The scores themselves never change; only the decisions do.

Before reading the curve, try the process by hand. Take four samples with true labels 1, 0, 1, 0. The classifier gives them the scores 0.9, 0.8, 0.7, and 0.1. The two samples labeled 1 are the positives, and the two labeled 0 are the negatives. The sample with score 0.9 is a positive whose score is highest, while the sample with score 0.8 is a negative that the classifier scores almost as high as the positive. Keep this small example in mind, because we will use it to trace the curve step by step.

The ROC curve plots the false positive rate on the horizontal axis against the true positive rate on the vertical axis. Each threshold produces one point, and the curve is the line connecting all of those points. Look at the four-sample example above. A threshold above 0.9 calls nothing positive, so both rates are zero and the point sits at the origin. Lowering the threshold to 0.9 catches the sample scored 0.9, which is a true positive, so the true positive rate rises while the false positive rate stays at zero.

Now lower the threshold to 0.8. The sample scored 0.8 is a negative, so the classifier has now produced its first false positive. Half of the actual positives are caught, and half of the actual negatives are wrongly called positive. The point moves right, to a false positive rate of 0.5, and stays level in true positive rate. At threshold 0.7 the second positive is caught, so the true positive rate rises to 1.0 while the false positive rate stays at 0.5. Finally, at threshold 0.1 every sample is called positive, which gives both rates a value of 1.0.

Each point is an operating point of the same classifier. The curve connects those points, so it describes what the classifier can do at every threshold rather than at one chosen cutoff. In general, moving the threshold from one extreme to the other traces the curve from the bottom left to the top right.

Both axes of the ROC curve are built from the four counts of a binary confusion table: true positives, false negatives, false positives, and true negatives. At any single threshold, each sample falls into exactly one of these four cells. The true positive rate divides true positives by the number of all actual positive samples. The false positive rate divides false positives by the number of all actual negative samples.

The true positive rate is exactly recall, the fraction of all actual positives the classifier catches. The false positive rate is the fraction of all actual negatives that were wrongly called positive. Because FPR divides by the number of negatives, it measures how much cost the false positives impose relative to the size of the negative class. A model can have a high TPR while still producing many false alarms, and FPR is what exposes that.

The ideal ROC curve sits close to the top left corner, where the true positive rate is high and the false positive rate is low. In that corner the classifier finds most of the positives while producing few false alarms. A curve that instead hugs the diagonal from bottom left to top right means the true and false positive rates are rising together, which is close to guessing. The further the curve bends toward the top left, the better the classifier separates the two classes across thresholds.

The curve also suggests a practical question. Compared with the default threshold of 0, the source's curve shows that a noticeably higher recall, around 0.9, can be reached while the false positive rate rises only a little. The point closest to the top left might therefore be a better operating point than the default. This is a suggestion to examine, not a guarantee, and it leads directly to the question of how the threshold should be chosen.

The scikit-learn function roc_curve returns three arrays: the false positive rate at each threshold, the true positive rate at each threshold, and the thresholds themselves. Each index describes one operating point. The function roc_auc_score returns the area under the curve as a single number. Here is the four-sample example from earlier computed in code.

The output of 0.75 can be checked by hand. The area under the curve is made of two trapezoid-shaped pieces: a rectangle of width 0.5 and height 0.5, giving 0.25, and a rectangle of width 0.5 and height 1.0, giving 0.5. Together they make 0.75. The same number also has a ranking meaning, explained in the next section.

In this formula, s+ is the score of a randomly chosen positive sample and s- is the score of a randomly chosen negative sample. AUC is the probability that a random positive receives a higher score than a random negative. Check this on the example: there are four positive-negative pairs. The pairs (0.9, 0.8), (0.9, 0.1), and (0.7, 0.1) are ranked correctly, while (0.7, 0.8) is not. Three correct pairs out of four gives 0.75. Because this ranking statement does not depend on any single threshold, AUC summarizes performance across all thresholds together.

The source tests the idea on a harder case: classifying all nines in the digits dataset against all other digits. This is an imbalanced problem, because nines are a small share of the samples. An SVM is trained with three different settings of the kernel bandwidth, gamma, and each setting gives an accuracy of 90%. Accuracy alone would suggest the three models are equivalent. The source notes that this could be chance-level performance, or it might not be.

The AUC and the curves separate the three models clearly. With gamma=1.0 the AUC is at chance level, so the decision function is no better than random. With gamma=0.01 the AUC is 1.0, meaning every positive sample is ranked above every negative sample. In the source's words, with the right threshold this model can classify the data perfectly, and adjusting the threshold would yield strong predictions. Accuracy alone would never have revealed that difference.

The source also compares two different model types on the same test data. It computes an AUC for a random forest, using predicted probabilities for the positive class, and an AUC for an SVC, using the decision function. roc_auc_score only needs a continuous score that ranks samples, so either works. On this data the random forest scores 0.937 and the SVC scores 0.916, so the random forest performs noticeably better by this measure.