Overfitting and Underfitting

The Training-Set Trap

A model can become better at the examples it has already seen while becoming less dependable on new examples. This happens because fitting familiar data is easier than learning patterns that generalize beyond it. The central challenge is not to make the model learn less carefully. It is to prevent the model from using its flexibility to memorize training-specific details.

Capacity and Memorization

Model capacity is controlled through the number of learnable parameters. A model with too little capacity may fail to capture relevant patterns and is underfit. As capacity increases, the model can represent more useful structure. If it has too much flexibility, it can also use that flexibility to memorize training examples and training-specific details. The target is a balance between underfitting and overfitting.

The visual shows why increasing capacity is not automatically beneficial. Too little capacity leaves relevant patterns unrepresented. Balanced capacity supports useful representations. Excessive capacity can make unrestricted memorization easier.

Three Training Checkpoints

This progression is important because overfitting is not always visible from training performance alone. At first, training and validation behavior often improve together. Later, optimization can continue while generalization stops improving.

Optimization Versus Generalization

Optimization and generalization measure different kinds of model performance. Reducing training loss is evidence that optimization is continuing, but it does not prove that the model is learning more useful behavior for unseen data. Validation performance helps reveal whether continued training is still improving generalization.

Pressure Against Memorization

When more data cannot be obtained, overfitting can be addressed by changing how much information the model can store or by placing constraints on the information it is allowed to store. The main controls in this lesson are model size, weight regularization, and dropout. They work differently, but all make accidental memorization less attractive than learning representations that retain predictive value.

Weight-Based Costs

L1 and L2 regularization add different weight-based costs to the loss function. The regularizer is attached to a layer's kernel weights, so the layer contributes an additional cost based on those weights. This leaves the model architecture in place while making certain weight choices less attractive during learning.

Dropout Changes by Phase

Dropout changes layer outputs during training by randomly zeroing features. Because a different subset of features can be removed for different training examples, the network is discouraged from depending on a fragile combination of features that only works for particular training cases. This breaks up happenstance patterns that the network might otherwise memorize.

Inference uses the full network differently: units are not dropped, and outputs are scaled rather than being produced by randomly removing features. Therefore, training and inference do not present exactly the same output behavior to the model.

Selecting a Remedy

Start by comparing training and validation behavior. If more training data is possible, it is the strongest general remedy because additional examples make generalizable patterns more likely. If more data is not available, reduce memorization resources or constrain what the model can store. Reducing model size changes the number of parameters. Regularization keeps the architecture but adds a weight-based cost. Dropout changes which output features are available during training.

Common Misreadings

• Treating lower training loss as proof that generalization is improving.

  Optimization and generalization measure different kinds of performance.

  Fix: Inspect validation performance alongside training performance.

• Calling every poorly performing model overfit.

  That describes underfitting, not overfitting.

  Fix: Ask whether useful progress is still missing or whether training-specific details have begun to dominate.

• Assuming that reducing overfitting means making the model learn less carefully.

  The goal is to prevent memorization and encourage useful representations, not to abandon careful learning.

  Fix: Describe the intervention as changing the pressure on learning.

• Treating dropout as a permanent reduction in model size.

  Dropout changes output information during training rather than permanently removing parameters.

  Fix: Distinguish training-time feature removal from inference-time scaled full-network outputs.

• Treating L1 and L2 regularization as the same mechanism.

  The source distinguishes them as different costs added to the loss function.

  Fix: Remember that both constrain weights through the loss, while L1 and L2 are not identical penalties.

Check Your Reasoning

Key Takeaways

1. Underfitting means that the model has not captured all relevant patterns in its training data.
2. Overfitting begins when the model learns training-specific details that do not transfer reliably to new data.
3. Optimization concerns fitting the training data, while generalization concerns performance on unseen data.
4. Model size, weight regularization, and dropout reduce overfitting through different mechanisms.
5. Validation performance reveals when continued optimization has stopped improving generalization.