Optimizers and Learning Rate
A Keras callback is an object passed to fit that is called at various points during training.
Why Training Needs Oversight
When training begins, you usually cannot know in advance how many epochs will be needed to reach the best validation loss. Training for a fixed, very long time may allow overfitting; training for too few epochs may stop before the model has improved enough. Keras callbacks provide a way to observe training as it happens and respond without requiring a separate trial run from scratch.
A Keras callback is an object passed to fit that is called at various points during training.
Training Event Handoffs
The training loop remains responsible for carrying out training, but control can move to callback behavior at selected points. These points can occur around training as a whole, around an epoch, or around a batch. At such an event, a callback can observe the current training situation and perform the action it was designed for. After the callback response, training can continue unless the callback's purpose is to interrupt it.
The important idea is not that every callback acts at every event. A callback is designed to respond at selected training events, so its behavior depends on both its purpose and the information it monitors.
Three Built-In Responses
| Callback | Main question | Response |
|---|---|---|
| ModelCheckpoint | Should important model weights be preserved? | Saves weights during training and can keep only the best model according to a monitored metric. |
| EarlyStopping | Is continuing to train no longer useful? | Interrupts training when a monitored metric stops improving for the configured patience period. |
| ReduceLROnPlateau | Has validation loss stopped improving, but should training continue more cautiously? | Changes the learning rate when validation loss stops improving. |
These callbacks do not solve the same problem. ModelCheckpoint protects a useful result by preserving weights. EarlyStopping changes the duration of training by interrupting it. ReduceLROnPlateau changes the optimizer's learning rate when validation loss has stopped improving, allowing the training process to respond without immediately stopping.
Metric Monitoring and Patience
A monitored metric gives a callback a signal about training behavior. The callback compares the current metric behavior with earlier behavior and waits according to its configured patience. If the monitored metric continues to improve, the callback does not treat the situation as a plateau. If improvement stops for the configured patience period, the callback takes its particular action: preserving weights, interrupting training, or changing the learning rate.
A Plateau During Validation
A training run monitors validation loss. The loss improves for several epochs, then stops improving. Which callback response matches each goal?
Preserve the best result: Use ModelCheckpoint so weights can be saved during training and the best model according to the monitored metric can be kept.
Stop when more training is no longer useful: Use EarlyStopping with a configured patience period. If the monitored metric stops improving for that period, training is interrupted.
Continue with a changed learning rate: Use ReduceLROnPlateau when validation loss stops improving and the desired response is to change the learning rate rather than immediately stop.
The same plateau can support different actions. The correct callback depends on whether the goal is preserving weights, ending training, or changing the learning rate.
Learning-Rate Adjustment
ReduceLROnPlateau is specifically intended for the case where validation loss has stopped improving and changing the learning rate is preferable to immediately stopping training. The callback responds to the observed plateau by changing the learning rate, so subsequent training proceeds under the adjusted learning-rate setting.
ReduceLROnPlateau and EarlyStopping can both react to a lack of improvement, but their outcomes differ. ReduceLROnPlateau changes the learning rate; EarlyStopping interrupts training.
Custom Callback Behavior
Built-in callbacks do not cover every possible training action. A custom callback is created by subclassing keras.callbacks.Callback and implementing methods that run at selected training events. This lets the callback define its own response to the information available at those events instead of using only the predefined save, stop, or learning-rate behaviors.
Choose the callback event and response together. First identify when the needed information becomes available; then implement the custom callback method for that event. This keeps the callback focused on one training-time action.
Common Selection Mistakes
Using EarlyStopping when the actual goal is only to preserve the best weights.
EarlyStopping interrupts training; it does not primarily provide the weight-preservation behavior described for ModelCheckpoint.
Fix:
Use ModelCheckpoint when preserving weights according to a monitored metric matters.Using ModelCheckpoint when the goal is to end unproductive training.
Saving weights does not by itself interrupt training.
Fix:
Use EarlyStopping with an appropriate patience period.Treating ReduceLROnPlateau as an immediate stop mechanism.
ReduceLROnPlateau changes the learning rate rather than immediately interrupting training.
Fix:
Use ReduceLROnPlateau when changing the learning rate is the desired response.Ignoring the monitored metric and patience configuration.
Callback behavior depends on the metric being monitored and the configured patience period.
Fix:
Decide which metric represents useful progress and how long to wait before acting.Assuming built-in callbacks cover every training action.
Built-in callbacks do not cover every possible training action.
Fix:
Create a custom callback by subclassing keras.callbacks.Callback and implementing methods for selected training events.
Choose the Response
A model's validation loss has stopped improving. You want to keep a record of the best model, stop training if the lack of improvement continues, and change the learning rate if the plateau should be handled without immediately stopping. Match each goal to the most appropriate callback: preserving the best weights, interrupting training, and changing the learning rate.
Hints
- Ask whether the goal is to preserve weights, interrupt training, or adjust the learning rate.
- Patience controls how long a callback waits after the monitored metric stops improving.
Checking Your Selection
Match each training goal with ModelCheckpoint, EarlyStopping, or ReduceLROnPlateau.
Preserve the best weights: The goal is saving weights during training and keeping the best model according to a monitored metric.
Interrupt unproductive training: The goal is stopping when a monitored metric stops improving for the configured patience period.
Continue after a validation-loss plateau: The goal is changing the learning rate when validation loss stops improving rather than immediately stopping.
The matches are ModelCheckpoint for preserving weights, EarlyStopping for interrupting training, and ReduceLROnPlateau for changing the learning rate.
Key Takeaways
- A Keras callback is an object passed to fit and invoked at selected points during training.
- ModelCheckpoint preserves important weights, EarlyStopping interrupts training, and ReduceLROnPlateau changes the learning rate after validation loss stops improving.
- The monitored metric and patience period determine when a callback recognizes insufficient improvement and acts.
- Custom callbacks extend training control by subclassing keras.callbacks.Callback and implementing methods for selected training events.
Key Takeaways
- Callbacks let Keras observe training and respond during the run instead of requiring a separate trial run.
- Choose ModelCheckpoint to preserve weights, EarlyStopping to interrupt unproductive training, and ReduceLROnPlateau to change the learning rate after a validation-loss plateau.
- A monitored metric supplies the progress signal, while patience determines how long the callback waits before acting.
- Custom callbacks provide training-event behavior beyond the built-in save, stop, and learning-rate responses.