Overfitting

The Familiar-Data Trap

A model can appear to improve while becoming less dependable. Its performance on the examples used for training may continue to get better, while its performance on data it has not seen stops improving or begins to decline. This is the central difficulty of overfitting: success on familiar examples is not the same as reliable performance on new examples.

Training Through Three Phases

Training often begins with underfitting. At this point, the model has not captured all the relevant patterns in the training data, so useful progress is still available. As training continues, performance on both the training examples and unseen examples can improve. After a certain point, however, validation performance may reach its strongest level. Continued training can still improve the training result while validation performance stalls and then worsens. That is the transition into overfitting.

Optimization and Generalization

Optimization and generalization describe different kinds of model performance. Optimization concerns improvement on the training examples. Generalization concerns how reliably the model performs on data it has not seen. These measures often move together at the beginning of training, but they do not have to continue moving together indefinitely.

Validation performance helps reveal when continued training has stopped improving generalization. If training performance keeps improving but validation metrics stop improving, the model is no longer gaining dependable performance on new data. If validation metrics then worsen, the model is moving further into overfitting.

Patterns That Transfer

A useful pattern captures regularities that help the model perform on new examples. A training-specific pattern is tied to details of the training data and does not provide reliable guidance outside that data. Overfitting begins when the model starts learning these training-specific details instead of concentrating on patterns that are more dependable for new data.

Augmenting the Training Collection

The original data is the starting point for the augmented collection. Transformations are applied to those existing examples, producing additional training examples. The result is an artificially larger dataset. Data augmentation is used to mitigate overfitting because a model can otherwise be trained on a dataset that does not contain enough variation for the learning task.

Reducing Overfitting

The strongest general remedy for overfitting is to obtain more training data. More data gives the model more examples from which to learn and makes it more likely that the model develops patterns that generalize. Data augmentation is one way to artificially increase the number of training examples when the existing collection does not contain enough variation.

When obtaining more data is not possible, other approaches change how much information the model can store or place constraints on the information it is allowed to store. Limiting what the model can retain makes unrestricted memorization less available. The optimization process is then pushed toward the most prominent patterns, which have a better chance of generalizing.

• Obtain more training data when possible.
• Use data augmentation to apply transformations to existing data and create additional training examples.
• Limit how much information the model can retain when more data is unavailable.
• Place constraints on the information the model is allowed to store so unrestricted memorization is less available.
• Monitor validation performance to identify when continued training stops improving generalization.

Common Misreadings

• Treating improved training performance as proof of improved generalization.

  Optimization and generalization measure different kinds of performance.

  Fix: Compare training behavior with validation behavior. Continued training improvement can coexist with worsening performance on unseen data.

• Calling every poorly performing model overfit.

  This describes underfitting, not overfitting.

  Fix: Use underfitting when useful patterns remain unlearned. Use overfitting when the model learns training-specific patterns that do not transfer reliably.

• Treating transformed examples as unrelated to the original data.

  Data augmentation applies transformations to existing data, and the original data is the starting point for the augmented collection.

  Fix: Recognize transformed examples as additional training examples produced from existing data.

• Assuming that more training always improves the model's usefulness on new data.

  Further training can make the model increasingly tailored to the training set.

  Fix: Watch validation performance for signs that generalization has stalled or worsened.

Check Your Understanding

Key Takeaways

1. Overfitting happens when a model learns training-specific patterns that do not transfer reliably to new data.
2. Optimization can continue improving training performance after generalization has stopped improving.
3. Underfitting can progress into useful generalization and then into overfitting as training continues.
4. Data augmentation applies transformations to existing data to create an artificially larger collection of training examples.
5. More data, data augmentation, limits on retained information, constraints on stored information, and validation monitoring can help reduce overfitting.