Training and Validation Loss
A Keras callback is an object passed to fit that is called at various points during training.
Why Observe Training
When training begins, you usually cannot know in advance how many epochs will be needed to reach the best validation loss. One approach is to train long enough for overfitting to begin, choose a suitable number of epochs, and then start another training run from scratch. That approach wastes time. Keras callbacks provide a way to observe training as it happens and respond during the run.
A Keras callback is an object passed to fit that is called at various points during training.
A callback turns training from a process that only runs toward a fixed endpoint into a process that can observe training behavior and respond while the model is fitting.
Callback Invocation Timeline
The model invokes callbacks at points during training. The exact response depends on the callback and the training information it monitors. For example, a callback can be designed to respond to a selected training event, inspect the current training information, and then save progress, interrupt training, change the learning rate, or perform another action.
The important idea is not that every callback performs an action at every listed point. Instead, a callback is invoked at training events relevant to its implementation. Built-in callbacks use those opportunities to inspect a monitored metric and decide whether their configured response should occur.
Metrics and Patience
A callback needs a monitored metric so that it can relate its response to training behavior. Training loss and validation loss are examples of the behavior that can be observed during fitting. A callback can use the monitored metric to determine whether the model is improving or whether progress has stopped.
Patience controls how long a callback waits after the monitored metric stops improving before it responds. The response depends on the callback: the configured patience period can influence when training is interrupted, when the learning rate is changed, or when a callback takes another configured action.
Selecting a Built-In Callback
The main selection question is what you want to change in response to training behavior. ModelCheckpoint is for preserving model weights. EarlyStopping is for interrupting training when a monitored metric has stopped improving for the configured patience period. ReduceLROnPlateau is for changing the learning rate when validation loss has stopped improving instead of immediately stopping training.
| Callback | Primary response | Use it when |
|---|---|---|
| ModelCheckpoint | Saves weights during training | Preserving model progress matters |
| EarlyStopping | Interrupts training | Continuing is no longer useful after the configured patience period |
| ReduceLROnPlateau | Changes the learning rate | Validation loss has stopped improving and training should adjust rather than immediately stop |
Choose the callback by the action you want it to take.
Choosing by desired action
A training run needs three different forms of control: preserve the best model according to a monitored metric, stop when the metric no longer improves, and change the learning rate when validation loss stops improving.
Preserve progress: Choose ModelCheckpoint because its purpose is to save weights during training and it can keep only the best model according to a monitored metric.
End unproductive training: Choose EarlyStopping because it interrupts training when a monitored metric stops improving for the configured patience period.
Adjust continued training: Choose ReduceLROnPlateau because it changes the learning rate when validation loss stops improving.
The correct sequence is ModelCheckpoint, EarlyStopping, and ReduceLROnPlateau. The decision comes from the intended response: save, stop, or change the learning rate.
Custom Training Responses
Built-in callbacks do not cover every possible training action. When the required behavior is not saving weights, interrupting training, or changing the learning rate in the ways provided by the built-in callbacks, a custom callback can provide the missing response.
A custom callback is created by subclassing keras.callbacks.Callback and implementing methods that run at selected training events.
Because the callback is connected to the training process, its event methods can respond when the selected events occur. A custom callback can also access model information, including information associated with the model, training logs, metrics, or optimizer, when that information is available to the callback during the event.
Think of a custom callback as a listener attached to training. The training loop reaches a selected event, the callback method is invoked, and the method uses the information available at that point to perform its custom behavior. The callback is therefore not a separate training run; it is a response mechanism inside the current run.
Mistakes Beginners Make
Choosing a callback without identifying the desired response.
EarlyStopping interrupts training, while ModelCheckpoint saves weights during training.
Fix:
Decide first whether the response should be saving, stopping, or changing the learning rate.Treating patience as a metric.
Patience is a configured waiting period used after the monitored metric stops improving.
Fix:
Separate the monitored metric from the amount of time the callback waits before responding.Assuming that a built-in callback covers every possible action.
Built-in callbacks do not cover every possible training action.
Fix:
Create a custom callback by subclassing keras.callbacks.Callback and implementing methods for selected training events.Ignoring validation behavior when choosing ReduceLROnPlateau.
ReduceLROnPlateau changes the learning rate when validation loss stops improving.
Fix:
Use validation loss as the relevant behavior when the goal is to adjust the learning rate after a plateau.
Apply the Decision Rule
Choose ModelCheckpoint, EarlyStopping, ReduceLROnPlateau, or a custom callback for each situation. First, you want to keep the best model according to a monitored metric. Second, you want training to stop after a monitored metric fails to improve for the configured patience period. Third, validation loss has stopped improving, and you want to change the learning rate rather than immediately stop. Fourth, you need a training action not covered by the built-in callbacks.
Hints
- Match each situation to the intended action: save, stop, change the learning rate, or implement a new response.
- The monitored metric and patience wording identify the control behavior, but the desired action identifies the callback.
The answers are ModelCheckpoint, EarlyStopping, ReduceLROnPlateau, and a custom callback, respectively.
Key Takeaways
- A Keras callback is an object passed to fit and invoked at points during training.
- Callbacks allow training behavior to be observed and acted on without requiring a separate trial run.
- ModelCheckpoint preserves weights, EarlyStopping interrupts training, and ReduceLROnPlateau changes the learning rate when validation loss stops improving.
- The monitored metric describes the behavior being observed, while patience determines how long a callback waits before responding.
- Custom callbacks subclass keras.callbacks.Callback, implement methods for selected training events, and can access model information available during those events.