Overfitting in Model Training
A falling training loss is necessary evidence that the model is fitting its training data, but it is not sufficient evidence that the model is good. Whether it is good depends on how it behaves on data it has not seen.
Try it: Training a Keras-Style Model: History and Early Stopping
The Keras workflow on a tiny dense network: stack Dense layers (the summary counts units * (inputs + 1) parameters per layer), compile with an optimizer, loss and accuracy metric, fit for a number of epochs in mini-batches with validation data, and read the history. Training loss keeps falling while validation loss reaches its lowest point and then rises: the model starts to overfit, and EarlyStopping with patience can stop there and restore the best weights.
How it works
- Build a Sequential stack of Dense layers ending in Dense(1, sigmoid); each layer has units * (inputs + 1) weights and biases, initialised Glorot-uniform with zero biases.
- compile(): mini-batch SGD with the chosen learning rate, binary cross-entropy (or mean squared error) loss, and accuracy (prediction > 0.5) as the metric.
- fit(): each epoch walks the training set in batches of batch_size; every batch runs a forward pass, backpropagates the mean batch loss and updates every weight .
- After each epoch the history records the epoch's training loss and accuracy (batch-weighted average of values seen during the epoch) and the val_loss and val_accuracy on the held-out validation points.
- EarlyStopping(monitor="val_loss", patience) keeps the best val_loss; after patience epochs with no improvement it stops training, and restore_best_weights puts back the weights of the best epoch.
- evaluate() reports loss and accuracy on the validation data with the final weights.
Default run (153 steps): Sequential([Dense(16, "relu"), Dense(16, "relu"), Dense(1, "sigmoid")]) on inputs of shape (2,). model.summary(): dense 16 * (2 + 1) = 48; dense_1 16 * (16 + 1) = 272; dense_2 1 * (16 + 1) = 17; total 337 trainable parameters, initialised with Glorot uniform (biases 0). … model.evaluate(x_val, y_val) with the weights from epoch 150: loss 1.2082, accuracy 0.75. Training loss went 0.6964 -> 0.0194. Stopping at epoch 32 would have given val_loss 0.5272 and val_accuracy 0.775.
Simplified: A toy re-implementation in TypeScript, not Keras or TensorFlow: 2 input features, at most 3 hidden layers of up to 32 units, a seeded 2-D dataset (inside/outside a circle with label noise; 40 validation points), plain SGD without momentum, shuffle=False so batch order is fixed, float64 arithmetic, and binary cross-entropy computed from the output logit (same value as on probabilities, without clipping). The results match an independent NumPy implementation of the same procedure, not a real Keras run.
Loading the simulation…
Two Kinds of Evidence from One Run
Once a model has been trained, the obvious question is whether the training actually worked. Reading what happened during training and deciding what it tells you is a practice sometimes called training results analysis. The key point is that a single training run produces more than one kind of evidence, and those pieces of evidence can point in different directions. A model can look successful by one measure while becoming less useful by another.
This article starts with a simple question: what does a falling training loss actually prove, and where does its meaning stop? Answering it carefully leads directly to overfitting, the condition in which a model becomes too specialized to its training data and performs poorly on new data.
What a Falling Training Loss Proves
Training set loss measures how far the model's outputs are from the target values on the data the model is learning from. When that loss is being driven down, the model is getting closer to those targets on that data.
The source treats the minimization of training set loss as evidence that the function is doing its job. That is a useful reading, because it tells you the model is learning something about the examples it was shown. It is a claim about the training data specifically.
Notice how carefully the claim is limited. A decreasing training loss is described as showing that the function is working. It is not described as showing that the model is finished, that it is accurate in general, or that it will succeed when it is used. The source keeps the evidence attached to the training set. That restraint is the first habit of good results analysis: say exactly which data a measurement was taken on.
A falling training loss is necessary evidence that the model is fitting its training data, but it is not sufficient evidence that the model is good. Whether it is good depends on how it behaves on data it has not seen.
Watching Loss and New-Data Performance Diverge
The source identifies a second thing happening during the same run. Even while the training set loss is falling, the model begins to overfit after a few epochs. An epoch is one full pass through the training data. The source ties this behavior to two conditions: the dataset is very small, and training iterates over it many times.
Read the diagram as one training run moving forward in time. The training loss does not have to stop falling for performance on new data to get worse. Once overfitting has begun, the two signals separate: the training measurement keeps improving while the measurement on new data moves the other way. The divergence is the thing to watch for, not any single number.
How a Model Becomes Too Specialized
The word specialized is the heart of the idea. A model that has overfit has not simply learned the general pattern in the data. It has fitted itself closely to the particular examples it was shown, so its behavior is tuned to those examples. New data does not match the training examples exactly. When the model meets it, the tuning stops helping and performance drops.
The conditions the source names are a very small dataset and many passes over it. A small dataset gives the model fewer varied examples to generalize from, and repeated iteration gives it more chances to adjust itself to those particular examples. Together they explain why the model can begin to overfit after only a few epochs.
Training Data Versus New Data
A good results analysis compares performance on training data with performance on new data, and it keeps each claim tied to the data it was measured on. The two comparisons answer different questions, so they should be read side by side rather than merged into one verdict.
| Question | Training data | New data |
|---|---|---|
| Which data was measured? | Data the model learned from | Data the model has never met |
| What does a good score show? | The function is working on that data | Whether the model performs in use on unseen cases |
| What can go wrong? | Loss can keep falling even after overfitting begins | Performance can drop while training loss improves |
Two measurements from the same run answer two different questions.
Following One Small Training Run
A small dataset trained for many epochs
A model is trained on a very small dataset of a few examples, and training passes over that same dataset many times. At what point should you suspect overfitting, and which measurements tell you?
- 1Identify the conditions
The dataset is very small and training iterates over it many times. The source names both as conditions that lead the model to begin overfitting.
- 2Track the training loss
As epochs pass, the training set loss keeps falling. This is evidence that the function is fitting the examples it was shown, but it says nothing yet about new data.
- 3Add a measurement on new data
Evaluate the model on data it has never seen. If performance on that data is weakening while training loss still falls, the model has begun to overfit.
- 4Read the two results separately
Report the training loss as a fact about the training data and the new-data result as a fact about unseen data. Do not combine them into a single claim that the model is good.
A falling training loss alone would look like success. Only the measurement on new data reveals that the model has become too specialized to its training examples.
Mistakes Beginners Make with Training Loss
Common mistake Treating a falling training loss as proof that the model is finished
What it looks likeA run shows the loss dropping epoch after epoch, so the model is declared ready for use.
Why it failsThe source describes a falling training loss as evidence that the function is working on its training data. It does not say the model will succeed on new data.
Do this insteadKeep the claim tied to the training data, and check performance on data the model has never met before drawing a conclusion.
Common mistake Assuming that more epochs always help
What it looks likeTraining continues for many more passes over a very small dataset because the loss is still going down.
Why it failsThe source ties overfitting to a very small dataset iterated over many times, with overfitting beginning after a few epochs.
Do this insteadWatch the measurement on new data across epochs, and stop or adjust training when that measurement stops improving.
Common mistake Merging the training result and the new-data result into one verdict
Why it failsThe two measurements answer different questions, so combining them hides the divergence that signals overfitting.
Checking Your Reading of a Training Run
What do you think happens?
A model's training set loss keeps falling across many epochs. What happens to its performance on new data?
Reveal answer
It can drop even though the training loss is still falling
The source states that the model begins to overfit after a few epochs even while the training set loss is falling, and that overfit behavior makes performance on new data drop.
A colleague says, 'The training loss went to its lowest value, so the model works.' Explain what the training loss does and does not show, and describe the additional evidence you would need before accepting that the model works on new data.
Hints
- Ask which data the loss was measured on.
- Think about what a model fitted closely to particular examples does when it meets data that does not match them exactly.
Key Takeaways
- A falling training set loss is evidence that the function is working on the data it was trained on, but it does not by itself show how the model will perform on new data.
- Overfitting is the condition in which a model becomes too specialized to its training data and performs poorly on new data.
- The source links overfitting to a very small dataset iterated over many times, with the model beginning to overfit after a few epochs even while training loss keeps falling.
- Good results analysis compares performance on training data with performance on new data and keeps each claim tied to the data it was measured on.