Overfitting and Model Complexity

Imagine a classifier that reports it is very certain about a prediction. It is tempting to conclude that the prediction is probably right, and that the more certain the model sounds, the more trustworthy it is. This reasoning feels sensible, but it quietly mixes two separate questions. The first question is what the model says about its own confidence. The second question is how often the model is actually correct when it says that. This article is about the gap between those two questions and about the property that closes that gap.

The source ties the reliability of a model's uncertainty to its complexity, and it makes two claims that point in opposite directions. First, a model that is more overfitted tends to make more certain predictions, even when those predictions are wrong. Second, a model with less complexity usually has more uncertainty in its predictions. Both claims matter, and by the end of this article you should be able to explain why each one matters for calibration.

When a model reports a confidence, that report is a statement about itself. The question of whether the report can be trusted is a different one. Trust depends on what happens across many predictions, not on how a single prediction sounds. The source makes this concrete by describing the property that connects stated confidence to outcomes.

Read the formula in words: among all the predictions where the model reports confidence $p$, the fraction that turn out correct should be $p$. Notice that the claim is about a group of predictions that share the same stated confidence. One prediction made at 70% confidence can be right or wrong, and calibration says nothing about that single outcome. Calibration describes what happens across many predictions made at that confidence level.

Classifiers that expose uncertainty estimates give a probability for each class. In a binary problem with two classes, those two probabilities add up to 1. Because they must sum to 1, exactly one of the classes will be above 50% certainty, and that is the class the model predicts. The predicted class is therefore the one carrying the larger share of the probability.

Everything so far has concerned binary classification. The source notes that the decision_function and predict_proba methods also work in the multiclass setting, where there are more than two classes. The source illustrates this on the Iris dataset, which is a three-class classification dataset. Both methods provide uncertainty estimates in binary and multiclass settings, but they report those estimates in different forms.

The key point for reading multiclass output is that the per-class values are the model's reported uncertainty about each class. The binary rule that the two probabilities sum to 1 is the special case of two classes. When you move to three classes, you read each class's value in the same spirit, and the predicted class is the one the model assigns the largest share of its uncertainty.

The source ties the reliability of a model's uncertainty to its complexity. An overfitted model tends to make more certain predictions, and that certainty can be wrong. The reason is that an overfitted model can look confident for the wrong reason. Its high certainty reflects how closely it has fit the training data, not necessarily how often it will be right on new inputs. So certainty alone cannot be read as calibration.

The opposite pattern appears on the other side. A model with less complexity usually shows more uncertainty in its predictions. The source also shows a gradient boosting model, where the decision boundary is drawn next to the predicted probabilities for class 1 over the same dataset. The boundaries in that visualization are much more well-defined, and the small regions where the model is uncertain are clearly visible. Those narrow uncertain bands sit where the probability for class 1 is near the middle, close to the boundary between the two classes.

The belief that certainty signals reliability goes wrong at one specific step. The flowchart below follows the reasoning. A model reports high certainty, and the belief adds that the prediction is probably right. The check that calibration requires is to ask how often predictions made at that stated confidence are actually correct. If that rate matches the stated confidence, the report is calibrated. If it does not, the belief has separated from what calibration requires.