Overfitting and Validation Loss
Validation loss is a measure of the model's performance on the validation dataset.
Try it: Training a Keras-Style Model: History and Early Stopping
The Keras workflow on a tiny dense network: stack Dense layers (the summary counts units * (inputs + 1) parameters per layer), compile with an optimizer, loss and accuracy metric, fit for a number of epochs in mini-batches with validation data, and read the history. Training loss keeps falling while validation loss reaches its lowest point and then rises: the model starts to overfit, and EarlyStopping with patience can stop there and restore the best weights.
How it works
- Build a Sequential stack of Dense layers ending in Dense(1, sigmoid); each layer has units * (inputs + 1) weights and biases, initialised Glorot-uniform with zero biases.
- compile(): mini-batch SGD with the chosen learning rate, binary cross-entropy (or mean squared error) loss, and accuracy (prediction > 0.5) as the metric.
- fit(): each epoch walks the training set in batches of batch_size; every batch runs a forward pass, backpropagates the mean batch loss and updates every weight .
- After each epoch the history records the epoch's training loss and accuracy (batch-weighted average of values seen during the epoch) and the val_loss and val_accuracy on the held-out validation points.
- EarlyStopping(monitor="val_loss", patience) keeps the best val_loss; after patience epochs with no improvement it stops training, and restore_best_weights puts back the weights of the best epoch.
- evaluate() reports loss and accuracy on the validation data with the final weights.
Default run (153 steps): Sequential([Dense(16, "relu"), Dense(16, "relu"), Dense(1, "sigmoid")]) on inputs of shape (2,). model.summary(): dense 16 * (2 + 1) = 48; dense_1 16 * (16 + 1) = 272; dense_2 1 * (16 + 1) = 17; total 337 trainable parameters, initialised with Glorot uniform (biases 0). … model.evaluate(x_val, y_val) with the weights from epoch 150: loss 1.2082, accuracy 0.75. Training loss went 0.6964 -> 0.0194. Stopping at epoch 32 would have given val_loss 0.5272 and val_accuracy 0.775.
Simplified: A toy re-implementation in TypeScript, not Keras or TensorFlow: 2 input features, at most 3 hidden layers of up to 32 units, a seeded 2-D dataset (inside/outside a circle with label noise; 40 validation points), plain SGD without momentum, shuffle=False so batch order is fixed, float64 arithmetic, and binary cross-entropy computed from the output logit (same value as on probabilities, without clipping). The results match an independent NumPy implementation of the same procedure, not a real Keras run.
Loading the simulation…
Why Training Length Is a Decision
When you fine-tune a model, you have to decide how many times it will pass over the training data. That decision is made before training begins, and it shapes the result. Train for too few passes and the model may not have learned enough. Train for too many and it may start memorizing the training examples instead of learning patterns that carry over to new data.
This article shows how to make that choice by reading the training signals the model gives you while it trains. The main signal is validation loss, which tells you how the model performs on data it did not train on. Before we can read that signal, we need to be precise about the unit that the setting controls: the epoch.
What One Epoch Means
An epoch is one full pass through the training dataset. The number of epochs is the number of such full passes the training run makes.
Think of an epoch as a unit of progress through the data. After epoch 1, the model has seen every training example once. After epoch 2, it has seen every training example a second time. Each epoch revisits the same examples, so the count tells you how many times the model has had the chance to learn from the whole dataset.
Notice that the dataset does not change between epochs. What changes is the model's learned weights, which are adjusted as training proceeds. The epoch is the outer loop that counts how many times the whole dataset has been used.
Epochs as a Hyperparameter
The number of epochs is a hyperparameter. A hyperparameter is a setting you choose, rather than something the model learns on its own from the data. The model's weights are learned during training, but the number of passes that training makes is set by you before training begins.
The source is explicit that the right number depends on two things: the dataset and how difficult the task is. There is no universal value that works for every fine-tuning job. A small, easy task may need few passes, while a harder task may need more. The same number that suits one dataset may be too many or too few for another.
Reading Validation Loss Against Training Loss
Two signals drive the decision about epochs: whether the model is overfitting, and whether validation loss still has room to improve. Validation loss is a measure of the model's performance on the validation dataset. It is used to tell whether the model is overfitting or underfitting.
The validation dataset is data the model did not train on. That is why it matters. Training loss can keep falling simply because the model is getting better at the examples it has already seen. Validation loss shows whether that improvement carries over to examples it has not seen. When the two curves move together, the model is generalizing. When they separate, you need to look more closely.
Read the diagram from left to right. Early on, both losses fall together, which is what healthy learning looks like. At the point where validation loss stops improving while training loss keeps falling, the two curves begin to separate. The widening gap is the signature of overfitting: the model keeps getting better at the training examples but worse at generalizing to new data.
Recognizing Overfitting in a Loss Plot
Overfitting occurs when the model performs poorly on new data because it has been fit too closely to the training data. In a loss plot, the sign to watch for is the model overfitting after the first few epochs. When that happens, the source advises reducing the number of epochs, since further training is working against how well the model generalizes.
Reading a Run That Overfits Early
A fine-tuning run is planned for 20 epochs. After the run, the loss plot shows that validation loss bottoms out around epoch 3 and then rises for the rest of the run while training loss keeps falling. Should the number of epochs be reduced, increased, or kept?
- 1Identify the overfitting sign
Validation loss rises after the first few epochs while training loss keeps falling. That divergence is the overfitting pattern.
- 2Check the rule for this pattern
The source says that if the model overfits after the first few epochs, the number of epochs should be reduced.
- 3Rule out the opposite case
Increasing epochs applies only when the validation loss trendline suggests it could still improve. Here it has stopped improving and is getting worse, so increasing is the wrong move.
- 4Decide the action
Reduce the number of epochs. Training past the point where validation loss turned upward only pushes the model toward memorizing the training examples.
Reduce the number of epochs, because the model overfits after the first few epochs and further training works against generalization.
Choosing Reduce, Increase, or Keep
Once you can read the two signals, the decision becomes a sequence of checks. Ask first whether the model is overfitting. If it is, reduce the number of epochs. If it is not, ask whether the validation loss trendline suggests that further training would still lower it. If so, increase the number of epochs. If neither holds, keep the current number.
Notice that the two actions point in opposite directions. Reducing epochs is the response to overfitting, and increasing epochs is the response to a validation trendline that could still improve. Keeping the count is the choice when neither signal is clear. The starting value of five epochs gives you a baseline to judge against.
Mistakes When Setting the Epoch Count
Common mistake Treating the epoch count as something the model learns
What it looks likeLeaving the number of epochs unset and assuming training will find the right length by itself.
Why it failsThe number of epochs is a hyperparameter. It is a setting you choose before training, not a weight the model adjusts.
Do this insteadChoose a starting value, such as five, and then adjust it based on the training signals.
Common mistake Using one universal number for every job
What it looks likeCopying the epoch count from a previous project regardless of the new dataset.
Why it failsThe source says the right number depends on the dataset and on how difficult the task is. There is no universal value.
Do this insteadUse the starting value as a baseline, then decide based on what this dataset and task show.
Common mistake Watching only training loss
What it looks likeSeeing training loss fall every epoch and deciding the run is going well.
Why it failsTraining loss can keep falling while the model overfits. Validation loss is evaluated on data the model did not train on, so it reveals whether the gains generalize.
Do this insteadCompare validation loss against training loss across epochs, and watch for the point where the curves separate.
Common mistake Increasing epochs when validation loss is already rising
What it looks likeAdding more epochs after validation loss has started climbing, in the hope that it will recover.
Why it failsA validation loss that rises after the first few epochs is the overfitting pattern, and the source advises reducing epochs in that case.
Do this insteadReduce the number of epochs. Increase only when the validation trendline suggests further training would still lower it.
Checking Your Reading of a Run
What do you think happens?
In a run, training loss keeps falling across epochs, but validation loss has started rising after epoch 3. What does this pattern indicate, and what should you do with the number of epochs?
Reveal answer
The model is overfitting, so reduce the number of epochs.
Validation loss rising after the first few epochs while training loss keeps falling is the overfitting pattern. The source advises reducing the number of epochs in that case, because further training works against generalization.
A run on a small dataset shows validation loss still steadily falling at epoch 5, with no sign of separation from training loss. Decide whether to reduce, increase, or keep the number of epochs, and explain which signal drove your choice.
Hints
- Start with the overfitting check: is there any sign of overfitting after the first few epochs?
- If not, look at the validation trendline. Does it suggest further training would still reduce validation loss?
Key Takeaways
- An epoch is one full pass through the training dataset, and the number of epochs is a hyperparameter you choose before training rather than a value the model learns.
- The right number of epochs depends on the dataset and the difficulty of the task. Five epochs is usually a good starting point, but there is no universal recommendation.
- Validation loss measures performance on data the model did not train on. When it rises while training loss keeps falling, the model is overfitting.
- If the model overfits after the first few epochs, reduce the number of epochs. If the validation trendline suggests it could still improve, increase the number of epochs. Otherwise, keep the current number.