Decision Boundaries in Classification

Suppose a classifier tells you it is very sure about a prediction. It is tempting to conclude that the prediction is probably right, and that the more certain the model sounds, the more you can trust it. This feels reasonable, but it quietly mixes two separate questions. The first question is what the model says about its own confidence. The second question is how often the model is actually correct when it says that. This article is about the gap between those two questions, and about the property that closes it.

Keep both questions in mind as you read. A classifier can produce a confidence number for every prediction, and that number may look authoritative. Whether the number deserves your trust is a separate matter, and the rest of this article shows how to tell the difference.

Think of a model's confidence as a claim it makes about each prediction. The claim is stated as a number, such as 70%. The question you can actually check is what happened to the predictions that carried that claim. Did the predictions made with 70% confidence turn out right about 70% of the time? Or did they turn out right much more often, or much less often?

Read the formula in words. Among all the predictions where the model reports confidence p, the fraction that turn out correct should be p. The vertical bar means 'given that,' so the left side is the probability of being correct, given the stated confidence. The equation states that this probability equals the stated confidence itself. Notice that the claim applies to a group of predictions that share the same stated confidence, not to one prediction at a time.

The diagram above is a generated illustration rather than a result from the source. It shows the two situations the definition distinguishes. In the calibrated case, the group of predictions stated at 70% is correct about 70% of the time. In the miscalibrated case, the same stated 70% is attached to a different observed rate, and that difference is the gap calibration is meant to rule out.

Classifiers that expose uncertainty estimates give a probability for each class. In a binary problem with two classes, those two probabilities add up to 1. Because they must sum to 1, exactly one class will be above 50% certainty, and that is the class the model predicts. The predicted class is therefore the one carrying the larger share of the probability.

Notice what this does and does not tell you. The 0.7 is the model's stated confidence in class 1. It identifies the predicted class, but whether 0.7 is calibrated is a separate question, answered only by checking how often predictions stated at 0.7 are actually correct.

The same reading strategy extends to problems with more than two classes. The source notes that the decision_function and predict_proba methods work in the multiclass setting, where there are more than two classes. It illustrates this on the Iris dataset, which is a three-class dataset. Both methods provide uncertainty estimates in both binary and multiclass settings, though they report those estimates in different forms.

For a multiclass probability output, each position in the vector corresponds to one class. To find the predicted class, locate the position holding the highest value, then read off the class that position maps to. The values below are generated for illustration and are not taken from the source.

In the illustration, position 1 holds the highest value, 0.75, so the predicted class is Class 1. The values in the vector are a probability for each class, and the predicted class is the one with the largest share. The source does not spell out the multiclass version of the sum-to-one property, so do not assume a particular total from the binary case. Check the values the model actually returns.

A decision boundary is the separator between the regions of input space that a model assigns to different classes. Moving across the boundary changes which class is predicted. The probabilities the model reports also change as you move, and that change is where uncertainty lives.

The source describes a gradient boosting model whose decision boundary is shown alongside the predicted probabilities for class 1 over the same dataset. In that visualization the boundaries are much more well-defined, and the small regions where the model is uncertain are clearly visible. Those narrow uncertain bands are where the probability for class 1 sits near the middle, which means they lie close to the boundary between the two classes.

The source also notes that in its example the classifier is relatively certain for most points, so most points sit comfortably on one side of 50%. That overall picture tells you how the model behaves, but it does not by itself tell you whether the certainty is deserved. Whether the reported uncertainty reflects real uncertainty in the data depends on the model and its parameters.

Calibration can be checked by following a sequence of steps that moves from individual predictions to a comparison at the group level. The diagram below traces that sequence. Each step follows from the definition: predictions are grouped by the confidence they report, and the observed correct fraction in each group is compared with the stated number.

The divergence happens at the last step. A model can look certain at the first step, with every prediction reporting high confidence, and still fail the comparison. The stated certainty stops matching observed correctness exactly when the group's correct fraction differs from the number the model reported.

The source ties the reliability of a model's uncertainty to its complexity, and it makes two claims that point in opposite directions. First, a model that is more overfitted tends to make more certain predictions, even if those predictions might be wrong. Second, a model with less complexity usually has more uncertainty in its predictions.

This matters for calibration because an overfitted model can look confident for the wrong reason. Its high certainty reflects how closely it has fit the training data, not necessarily how often it will be right on new inputs. Certainty alone therefore cannot be read as calibration. A calibrated model's stated confidence has to line up with how often it is correct, and an overfitted model can report high confidence while that alignment fails.