Evaluating Clustering Metrics
When a clustering algorithm returns an array of numbers, each number is a cluster label.
Why Matching Labels Feels Like the Right Test
A very common first instinct when checking a clustering result is to compare its labels with a known answer and count how many labels match. That instinct comes from classification. In classification, a predicted class either equals the true class or it does not, so counting matches is a meaningful score. Applied to clustering, the same instinct produces a wrong verdict.
This article follows one mechanism step by step. You will see how accuracy_score compares two lists, why that comparison breaks down for clusterings, and which metric measures what a clustering actually is. Along the way you will predict results before running them and find the exact point where a printed value disagrees with your expectation.
How accuracy_score Reads Two Lists
When a clustering algorithm returns an array of numbers, each number is a cluster label. If a reference array of true labels exists, it seems natural to ask whether the two arrays agree, element by element. accuracy_score does exactly that. It checks each position and reports the fraction of positions where the two values are identical.
The trouble is that the numbers in a clustering are arbitrary names. A cluster labeled 0 in one run might be labeled 1 in another run, with no change to which points are grouped together. accuracy_score has no way to know that two different integers name the same group. It only sees that the integers differ at that position.
The diagram lays out the two labelings side by side. Four of the five positions disagree, even though the grouping they describe is identical. Only position 4 agrees, and it agrees only because the number 2 happens to appear in both lists in the same place. The metric is not measuring grouping quality at all. It is measuring whether the integers line up.
Running the Comparison Script
The script below compares two labelings of the same five points using both metrics. The true labels put points 0 and 1 together, points 2 and 3 together, and point 4 alone. The predicted labels use the same three groups with different numbers. Before you read the output, make a prediction for each printed line.
What do you think happens?
Two labelings describe the same grouping, but the numbers differ. What do you think accuracy_score prints, and what do you think adjusted_rand_score prints?
Reveal answer
accuracy_score prints 0.2 and adjusted_rand_score prints 1.0. The second option describes this result.
accuracy_score only counts positions where the integers are identical, and just one of the five positions matches. adjusted_rand_score asks whether the same pairs of points are kept together, and here they are, so it reaches its maximum value.
0.2
1.0true_labelsandpred_labelsdescribe the same grouping: points 0 and 1 share a group, points 2 and 3 share a group, and point 4 is alonepred_labelsuses 1 wheretrue_labelsuses 0, and 0 wheretrue_labelsuses 1, so the numbers are swapped but the grouping is unchangedaccuracy_scorecompares the two lists position by positionadjusted_rand_scorecompares which pairs of points are kept together
from sklearn.metrics import accuracy_score, adjusted_rand_score
true_labels = [0, 0, 1, 1, 2]
pred_labels = [1, 1, 0, 0, 2]
print(accuracy_score(true_labels, pred_labels))
print(adjusted_rand_score(true_labels, pred_labels))- 1Line 1 imports the two metric functions that the comparison uses.
- 2Line 3 defines the reference labels, where points 0 and 1 share label 0, points 2 and 3 share label 1, and point 4 has label 2.
- 3Line 4 defines the predicted labels, which group the same points together but with the numbers 1, 0, and 2 instead.
- 4Line 6 calls
accuracy_score, which checks each position for identical integers. Only position 4 matches, so one of five positions agrees and the result is 0.2. - 5Line 7 calls
adjusted_rand_score, which looks at pairs of points rather than integers. Every pair kept together in one labeling is kept together in the other, so the result is 1.0.
0.2
1.0Finding the Line Where Expectation Breaks
To locate a divergence, compare each step of the script with your prediction and find the first line whose output disagrees with it. In this script, the grouping is the same from the first line onward, so the data and the grouping are not where the trouble starts. The first printed value that disagrees with a reasonable expectation is the accuracy line.
Tracing the first mismatch in the script
A learner expects both printed values to be 1.0 because the two labelings describe the same grouping. Which line produces a value that disagrees with that expectation?
- 1Check the label definitions
true_labelsandpred_labelsare defined on lines 3 and 4. They group the points identically, so no disagreement appears in the grouping itself. - 2Check the accuracy call
The
accuracy_scorecall on line 6 prints 0.2. This is the first printed value that differs from the expected 1.0. - 3Identify the assumption
The accuracy call assumes that a correct answer means the same integer appears at the same position in both lists. That assumption is false for clustering, because the integers are arbitrary names.
- 4Check the second call
The
adjusted_rand_scorecall on line 7 prints 1.0, which matches the expectation.
The divergence appears at line 6, the accuracy_score call. The data and the grouping are correct, and the mismatch comes from the assumption built into the accuracy comparison.
What adjusted_rand_score Compares Instead
The source names two suitable alternatives: adjusted_rand_score and normalized_mutual_info_score. Both are designed to compare a clustering with a reference grouping while ignoring the label names. The key idea is that the meaningful content of a clustering is which points share a group. The integer attached to each group is an arbitrary name.
true_labels = [0, 0, 1, 1, 2]Group named 0 holds points 0 and 1
pred_labels = [1, 1, 0, 0, 2]Same points held by the group named 1
| Question the metric asks | accuracy_score | adjusted_rand_score |
|---|---|---|
| What does it compare? | The integer at each position | Which pairs of points share a group |
| Does renaming groups change the score? | Yes, it can drop sharply | No, the score stays the same |
| Suitable for clustering evaluation? | No, it judges names | Yes, it judges grouping |
Both metrics compare two labelings, but they ask different questions.
When you choose among metrics, keep one question in mind: does the metric depend on which integer each point receives, or only on which points share a group? A metric that depends only on shared groups is the one that fits clustering evaluation.
Clustering Without a Reference
Everything above assumes a reference grouping exists to compare against. Often it does not. The source treats evaluating clustering without ground truth as a distinct challenge, separate from comparing against known labels. Without a reference, metrics such as adjusted_rand_score cannot be computed, because they need a second clustering to compare with.
This article does not cover the specific methods used when no reference exists. What it does establish is the lesson that carries over: whatever method you choose must judge the grouping, and it must not depend on arbitrary label names.
Common Mistakes When Evaluating Clusters
Common mistake Counting matching integers to judge a clustering
What it looks likeaccuracy_score([0, 0, 1, 1, 2], [1, 1, 0, 0, 2]) returns 0.2 even though the grouping is identical.
Why it failsCluster labels are arbitrary names, so matching integers says nothing about whether the same points share a group.
Do this insteadUse a metric that judges which points are grouped together, such as adjusted_rand_score or normalized_mutual_info_score.
Common mistake Reading a low accuracy as a poor clustering
What it looks likeA learner sees 0.2 and concludes the algorithm failed on five points.
Why it failsThe low number reflects the renaming of groups, not a difference in which points were grouped.
Do this insteadCheck whether the grouping matches before trusting the score, and compare with a metric that ignores label names.
Common mistake Assuming adjusted_rand_score works with no reference
What it looks likeCalling adjusted_rand_score on a single clustering with no second labeling to compare against.
Why it failsThe metric needs a second clustering to compare with, so there is nothing to score against.
Do this insteadRecognize that evaluating without ground truth is a separate challenge and needs a different approach.
Practice Choosing and Tracing
Two labelings of the same six points are [0, 0, 0, 1, 1, 1] and [2, 2, 2, 0, 0, 0]. Predict whether accuracy_score and adjusted_rand_score will each return 1.0, then run both and identify which result matches your prediction. If one does not, name the line where the disagreement begins.
Hints
- Check whether each position holds the same integer in both lists.
- Check whether the same pairs of points share a group in both lists.
- Compare your prediction line by line with the printed output.
Key Takeaways
accuracy_scorecompares labels position by position, so it treats renamed but identical groupings as wrong.- A cluster label is an arbitrary name; the meaningful content of a clustering is which points share a group.
adjusted_rand_scoreandnormalized_mutual_info_scorejudge the grouping rather than the label names, so they suit clustering evaluation.- To locate a divergence, compare each step of the script with your prediction and find the first line whose output disagrees.
- Evaluating clustering without ground truth is a separate challenge, because reference-based metrics need a second clustering to compare with.