The F-Score and Precision-Recall Trade-off

A common first instinct, when a classifier scores well, is to treat the accuracy number as a verdict. If a model is correct on most of its test examples, it seems reasonable to conclude that it works. In a binary problem that conclusion is at least easy to check, because there are only two outcomes to keep track of. Multiclass problems are different. When there are ten or more possible labels, a single fraction hides which labels the model handles well and which ones it struggles with.

Multiclass accuracy is defined the same way as before: it is the fraction of correctly classified examples. The problem appears when the classes are imbalanced. Imagine a three-class problem in which 85% of the points belong to class A, 10% belong to class B, and 5% belong to class C. Consider a model that has learned nothing useful and simply predicts class A for every point. Work through the consequences before reading on.

The lesson is not that accuracy is always worthless. It is that an accuracy value must be read alongside information about how the errors are distributed across classes. The tools in the rest of this article supply that information: the confusion matrix, the per-class precision, recall, and f-score, and the ways those per-class values are averaged into one number.

A confusion matrix places true labels in rows and predicted labels in columns. Each cell counts how many examples with a given true label received a given predicted label. The diagonal, where row and column match, holds the correct predictions. Every off-diagonal cell is a confusion: a true class being predicted as some other class. Reading the grid row by row tells you where each true class's examples went.

The grid below is a small generated example with 100 points, matching the 85, 10, and 5 class sizes from the imbalance example. Row A holds 85 true A points, row B holds 10, and row C holds 5. Read the nonzero off-diagonal cells: they are the confusions.

Read the grid this way. Row B has 10 true members, of which 5 are correct, 3 were predicted as A, and 2 were predicted as C. Column A, by contrast, collects 87 predictions, most of them correct, but also picks up one true C point and three true B points. The grid shows which class pairs are confused, and in which direction, something no single accuracy number reports.

The source applies this reading to a real task: classifying the ten handwritten digits in the digits dataset. A logistic regression model is trained on part of the data, predicts labels for the held-out test set, and then has its accuracy and confusion matrix printed. The reported accuracy is 95.3%, which the source describes as a sign that the model is doing pretty well. The matrix is where the more detailed story begins, and the source also shows the same counts as a colored heatmap with predicted labels on the horizontal axis, true labels on the vertical axis, and digit names as ticks. The heatmap is only a visual version of the grid, so learning to read the raw numbers first is worthwhile.

A confusion matrix is detailed but dense. Per-class precision and recall condense it into two numbers for each class, computed by treating that class as the positive class and every other class as negative. Precision asks whether a prediction of a given class can be trusted. Recall asks whether every true member of that class was found.

Here TP is the count of true positives for class c, FP is the count of examples predicted as c that belong to another class, and FN is the count of examples of class c that were predicted as something else. In the grid, the diagonal cell for class c supplies TP, the rest of its column supplies FP, and the rest of its row supplies FN.

The source makes the point that precision and recall are separate questions, using rows from the digit classifier's report. Class 0 has precision and recall both equal to 1, because no confusions involve that class. Class 7 has precision 1, meaning no other digit was mistakenly predicted as 7, yet its recall is lower at 0.94, meaning a few true 7s were missed. Class 6 has recall 1, because no digit 6 was missed, even though its precision is not perfect. A model can therefore be trustworthy in one direction and incomplete in the other.

A classification report condenses the confusion matrix into one row per class, with precision, recall, and f-score for each, and usually a support count showing how many true examples that class has. The source computes it with the classification_report function. The code below sets up the digits task so the report can be produced. The imports use the standard scikit-learn locations for each function involved.

Read the report one row at a time. Each row answers three questions for one class: how often a prediction of that class was right (precision), how often that class was found (recall), and the single balanced score combining them (f-score). The support column tells you how many true examples of that class were in the test set, which is the context needed to judge how much each row should matter.

The per-class rows are also where the model's trouble spots appear. In the digit report, class 8 has the lowest f-score at 0.91, which makes it the first row to inspect when the overall score needs explaining. The source also flags class 3 as a problem row, because its precision is lower than most of the others. These are the rows to examine before trusting the headline 95.3% accuracy.

For imbalanced multiclass data, the source describes the most commonly used metric as the multiclass version of the f-score. The method is to compute one binary f-score per class, treating that class as positive and all other classes together as negative. That produces one f-score for each class, and a final single number is obtained by combining those per-class values in one of three ways.

The source computes the micro and macro averages on the same test predictions using the f1_score function with its average argument set to each strategy. The source's version of that line was incomplete, so the corrected form is shown below.

The source gives a direct rule for choosing between the two most common options. If each sample matters equally, the source recommends the micro average f1-score. If each class matters equally, it recommends the macro average f1-score. The choice is about what the application is trying to protect, not about which number happens to be higher.

Apply the rule to the generated grid. If the business cares most about the common case and every individual prediction counts equally, micro's 0.91 is the relevant figure. If the rare classes matter as much as the common one, for example because missing class C is costly regardless of its size, macro's 0.707 reveals the weakness that micro hides.