Learning Rate Schedules in Deep Learning
Cosine decay changes the size of update steps. It does not change the network's architecture or its weights. The schedule controls how far each update moves, not what the network is.
Why One Fixed Step Size Is a Compromise
When a deep neural network or a large language model is trained, it repeatedly nudges its weights in a direction that should lower its loss. The loss is the measure training tries to reduce. The size of each nudge is controlled by the learning rate, so the learning rate sets how big each step through the weight space is.
If the learning rate stays fixed for the entire run, it becomes a single compromise. It must be large enough to make progress early on, yet small enough to avoid causing trouble later. A value that works well at the start of training can be too aggressive near the end, and a value that is gentle enough for the end can make the early stages painfully slow. Cosine decay is one technique that resolves this tension by letting the learning rate change over time instead of holding it constant.
Practitioners who train complex deep networks and large language models adopt cosine decay widely. This article explains what the schedule looks like, why its shape matters, and how to reason about it when you read a training configuration.
What the Learning Rate Changes at Each Step
The learning rate is not a fixed number chosen once at the start of training. It is a value that is recalculated as training moves forward, so that each epoch can run with a different step size than the one before it. Cosine decay is a technique that modulates the learning rate throughout the training epochs, making it follow a cosine curve after the warmup stage.
Cosine decay changes the size of update steps. It does not change the network's architecture or its weights. The schedule controls how far each update moves, not what the network is.
Where Cosine Decay Starts and Ends
Notice the phrase after the warmup stage. Cosine decay does not govern the whole run. Training first passes through a warmup stage, and only once that stage has finished does the learning rate begin tracing the cosine curve. The decay is therefore a second phase layered on top of an earlier one, not a replacement for it.
The diagram below traces the learning rate across a whole run. Read it from left to right. The warmup stage comes first, the handoff happens at the end of warmup, and the decay phase carries the learning rate down toward its final value at the last training step.
The handoff is the key boundary. Warmup ends, and the learning rate passes into the cosine curve at approximately the level warmup left it. From that point, the learning rate is still close to its warmup value, so updates remain fairly large at the start of the decay phase. The learning rate then falls, and at the end of training it has settled toward a value that is nearly zero.
Predicting the Shape of the Descent
Before you read the description of the curve again, try to form a prediction about its behavior. Consider a learning rate that starts at a meaningful value after warmup and must eventually become very small. The question is about the shape of that descent, not about exact numbers.
What do you think happens?
After warmup ends, how do you expect the learning rate to move until the final training step?
Reveal answer
It starts near the warmup level, drops fastest through the middle, and settles toward a value near zero by the end.
At the beginning of the decay phase the rate is still close to where warmup left it. Through the middle it drops at its quickest pace. Near the end it has settled toward a value that is nearly zero, so weights change only slightly from one epoch to the next.
Three Stages of the Decay Phase
Following the shape through a run, you can describe three stages. At the beginning of the decay phase, the learning rate is still close to where warmup left it, so updates remain fairly large. Through the middle, the learning rate is dropping at its quickest pace, so each epoch takes noticeably smaller steps than the last. Near the end, the learning rate has settled toward a value that is nearly zero, so the weights change only slightly from one epoch to the next.
The late-stage behavior is the purpose of the schedule. Because the learning rate is reduced to nearly zero, late-stage weight updates become very gentle.
How a Shrinking Step Prevents Overshooting
The loss is the measure training tries to reduce, and the lowest point of the loss is the loss minimum. When each weight update is large, the model can move so far in one step that it passes over the lowest point and lands on the other side, where the loss is higher again. This is called overshooting the loss minimum.
Cosine decay addresses this by reducing how far each update can move the weights. The source describes the purpose of the gradual decrease as decelerating the pace at which the model updates its weights. A slower pace leaves less room for a single step to carry the model past the minimum it is trying to settle into. The diagram below compares a large update near the minimum with a small one.
The source ties this to stability. Because the risk of overshooting is reduced, training stays steadier during its later phases. This matters most late in training, when the model is already close to a good solution and large jumps would be most damaging.
Tracing One Run from Warmup to the Final Step
Following one run from warmup to the final step
A training run uses a warmup stage, then cosine decay for the rest of training. Trace how the learning rate and the size of each update change from start to finish.
- 1Warmup stage
Cosine decay has not started yet. The warmup stage is still governing the learning rate.
- 2Handoff
When warmup finishes, the learning rate begins tracing the cosine curve from approximately the level warmup left it.
- 3Early decay
The learning rate is still close to its warmup value, so updates remain fairly large.
- 4Middle of decay
The learning rate is dropping at its quickest pace, so each epoch takes noticeably smaller steps than the last.
- 5Final steps
The learning rate has settled toward a value that is nearly zero, so weights change only slightly between epochs.
The size of each update is fairly large just after the handoff, shrinks steadily through the middle of the decay, and is smallest at the end of training.
Fixed Rate Versus Cosine Decay
The table below sets a fixed learning rate beside cosine decay. The difference is not in the architecture or the weights, which both approaches leave alone. The difference is in how the step size is allowed to change over the run.
| Aspect | Fixed learning rate | Cosine decay |
|---|---|---|
| Value over the run | One value held for the entire run | Recalculated as training moves forward, following a cosine curve after warmup |
| Early-stage step size | Must be large enough to make progress | Starts near the level warmup left it |
| Late-stage step size | Must still be small enough to avoid trouble | Settles toward a value that is nearly zero |
| Overshooting risk late in training | Remains with the single compromise | Reduced because late updates are very gentle |
Fixed learning rate compared with cosine decay
Misunderstandings About Cosine Decay
Common mistake Assuming cosine decay controls the whole training run
Why it failsCosine decay does not govern the whole run. Training first passes through a warmup stage, and the cosine curve begins only after warmup has finished.
Do this insteadLocate the end of warmup first, then treat the cosine phase as a second phase that starts at that handoff.
Common mistake Believing the learning rate drops to zero early in training
Why it failsThe early part of the decay phase still has a learning rate close to its warmup level. The rate settles toward a value that is nearly zero only near the end.
Do this insteadExpect large-to-moderate steps at the start of decay, a steep drop through the middle, and a value near zero at the final steps.
Common mistake Thinking cosine decay changes the network's architecture or weights
Why it failsCosine decay changes the size of update steps. It does not change the network's architecture or its weights directly.
Do this insteadDescribe the schedule as controlling how big each update is, not what the model is.
Common mistake Treating the learning rate as one number chosen once at the start
Why it failsThe learning rate is recalculated as training moves forward, so each epoch can run with a different step size.
Do this insteadWhen reading a configuration, think of the learning rate as a value that changes at every epoch.
Checking Your Understanding of the Schedule
A training configuration uses warmup followed by cosine decay. Explain in your own words why the learning rate near the final training step can be nearly zero without the model stopping its useful progress, and why this helps with overshooting the loss minimum.
Hints
- Think about where the cosine phase begins relative to warmup.
- Consider what happens to the size of each update when the learning rate is small.
- Recall why a large step can carry the model past the lowest point of the loss.
Key Takeaways
- Cosine decay modulates the learning rate throughout training so that it follows a cosine curve after the warmup stage.
- Cosine decay is a second phase layered on top of warmup: the decay begins at the handoff from warmup and does not govern the whole run.
- The learning rate starts the decay near its warmup level, falls fastest through the middle, and settles toward a value that is nearly zero at the end.
- A shrinking learning rate reduces how far each update can move the weights, lowering the risk of overshooting the loss minimum and supporting stability in later training.
- Cosine decay changes the size of update steps, not the network's architecture or weights.