Clustering Evaluation without Ground Truth

A very common first instinct when checking a clustering result is to compare its labels with a known answer and count how many labels match. That instinct comes from classification, where a predicted class either equals the true class or it does not. Applied to clustering, the same instinct produces a wrong verdict. This article follows that mistake step by step, shows exactly where the evaluation goes wrong, and replaces it with a metric that measures what a clustering actually is.

The lesson is best learned by tracing a small comparison script. Before each result is printed, you will be asked to predict it. The first printed result that disagrees with your prediction marks the place to examine most closely.

When a clustering algorithm returns an array of numbers, each number is a cluster label. If a reference array of true labels exists, it seems natural to ask whether the two arrays agree, element by element. accuracy_score does exactly that: it checks each position and reports the fraction of positions where the two values are identical.

The trouble is that the numbers in a clustering are arbitrary names. A cluster labeled 0 in one run might be labeled 1 in another run, with no change to which points are grouped together. The meaningful content of a clustering is which points share a group, not which integer each group received.

Consider five points. The first labeling puts points 0 and 1 in group 0, points 2 and 3 in group 1, and point 4 in group 2. The second labeling puts the same points together, but uses different names: points 0 and 1 get 1, points 2 and 3 get 2, and point 4 gets 0. Nothing about the grouping has changed. Only the names have been rotated.

Read the diagram position by position. Every position disagrees, so accuracy_score reports 0.0 even though the grouping is identical. The metric is not measuring whether the points are grouped correctly; it is measuring whether the same integer sits at the same position in both lists.

To locate the divergence, follow the script in order and compare each printed output with your prediction. The first line whose output disagrees with what you expected is the place to examine. The flowchart below lists the steps in execution order.

accuracy_score requires the exact names to match. For classification this is a reasonable demand, because the predicted class is meant to be the true class. For clustering, the integers are arbitrary, so requiring them to match asks the wrong question. A metric that rewards identical integers will penalize a grouping that is correct but named differently.

Each matched pair of groups holds exactly the same points, even though the numbers attached to them differ. A metric that asks only which points share a group sees identical clusterings here. accuracy_score asks which integer sits at each position, and sees none in agreement.

The source names two suitable alternatives: adjusted_rand_score and normalized_mutual_info_score. Both are designed to compare a clustering with a reference grouping while ignoring the label names. When choosing between them, keep one question in mind: does the metric depend on which integer each point receives, or only on which points share a group? A metric that depends only on shared groups fits clustering evaluation.

Everything above assumes a reference grouping exists to compare against. Often it does not. The source treats evaluating clustering without ground truth as a distinct challenge, separate from comparing against known labels.

The lesson that carries over is this: whatever method you choose must judge the grouping, and it must not depend on arbitrary label names. This article does not cover the specific methods used when no reference exists.