Loss Minimization and Gradient Descent
The loss is the measure that training tries to reduce.
Why One Fixed Step Size Falls Short
When a deep neural network or a large language model is trained, it repeatedly nudges its weights in a direction that should lower its loss. The loss is the measure that training tries to reduce. The size of each nudge is controlled by the learning rate.
A learning rate that stays fixed for the entire run is a single compromise. It must be large enough to make progress early in training, yet small enough to avoid causing trouble later. A value that suits the early stage can be too aggressive once the model is close to a good solution, and a value that suits the late stage can make early progress painfully slow.
Cosine decay is one technique that resolves this tension. Instead of holding the learning rate constant, it lets the learning rate change over time. Practitioners who train complex deep networks and large language models adopt it widely, and this article explains what its schedule looks like, why its shape matters, and how to reason about it when you read a training configuration.
Where Cosine Decay Begins and Ends
Cosine decay is a technique that modulates the learning rate throughout the training epochs, making it follow a cosine curve after the warmup stage. The key phrase is after the warmup stage. Cosine decay does not govern the whole run. Training first passes through a warmup stage, and only once that stage has finished does the learning rate begin tracing the cosine curve.
This means cosine decay is a second phase layered on top of an earlier one. It is not a replacement for warmup. The handoff between the two stages is the moment the learning rate stops following the warmup pattern and starts following the cosine pattern.
The schedule ends with the learning rate reduced to nearly zero. As a result, the weight updates in the late stage of training become very gentle. Cosine decay therefore covers the beginning of the decay phase, the middle, and the end, with the learning rate moving steadily from its post-warmup level down toward that near-zero value.
Tracing the Learning Rate Through the Decay
What do you think happens?
Across the decay phase, in which part does the learning rate fall most quickly: the start, the middle, or the end?
Reveal answer
The middle
Through the middle of the decay, the learning rate drops at its quickest pace, so each epoch takes noticeably smaller steps than the last. At the start of the decay it is still close to where warmup left it, and near the end it has settled toward a value that is nearly zero, so it changes only slowly at both ends.
The phrase modulates the learning rate throughout the training epochs means that the learning rate is not a fixed number chosen once at the start. It is recalculated as training moves forward, so each epoch can run with a different step size than the one before it.
Following that shape through a run, you can describe three stages. At the beginning of the decay phase, the learning rate is still close to where warmup left it, so updates remain fairly large. Through the middle, the learning rate is dropping at its quickest pace. Near the end, it has settled toward a value that is nearly zero, so the weights change only slightly from one epoch to the next.
How a Smaller Step Prevents Overshooting
The lowest point of the loss is the loss minimum, and it is the point training is trying to settle into. When each weight update is large, the model can move so far in one step that it passes over the lowest point and lands on the other side, where the loss is higher again. This is called overshooting the loss minimum.
Cosine decay addresses this by reducing how far each update can move the weights. It decelerates the pace at which the model updates its weights. A slower pace leaves less room for a single step to carry the model past the minimum it is trying to reach.
This matters most in the later phases of training. The model is already close to a good solution by then, so a large jump would be most damaging. Because the risk of overshooting is reduced, training stays steadier as it nears the end of the run.
Following One Run Stage by Stage
Reading one training run with cosine decay
A training run uses warmup followed by cosine decay. Describe what the learning rate and the weight updates are doing at the start of the decay, in the middle, and near the end.
- 1Identify the handoff
Warmup runs first. Cosine decay only begins after warmup finishes, so the decay phase starts from the level warmup left behind.
- 2Read the early decay
The learning rate is still close to its post-warmup level, so each update remains fairly large.
- 3Read the middle
The learning rate is dropping at its quickest pace here, so each epoch takes noticeably smaller steps than the one before.
- 4Read the end
The learning rate has settled toward a value that is nearly zero, so the weights change only slightly from one epoch to the next, which keeps late training gentle.
Cosine decay starts after warmup with fairly large updates, shrinks the steps fastest through the middle, and finishes with very gentle updates near a nearly zero learning rate.
Misconceptions About Cosine Decay
Common mistake Believing cosine decay controls the whole training run
What it looks likeReading a configuration with warmup and cosine decay and assuming the cosine curve is active from the very first epoch.
Why it failsTraining first passes through a warmup stage, and the learning rate only begins tracing the cosine curve once that stage has finished.
Do this insteadTreat cosine decay as a second phase that begins after warmup ends, layered on top of the earlier stage.
Common mistake Thinking cosine decay replaces warmup
Why it failsThe decay is a second phase layered on top of an earlier one, not a substitute for it.
Do this insteadKeep both stages in mind: warmup first, then cosine decay.
Common mistake Thinking the learning rate is a fixed number chosen once at the start
Why it failsCosine decay modulates the learning rate throughout the epochs, recalculating it as training moves forward.
Do this insteadExpect each epoch to potentially run with a different step size than the one before it.
Common mistake Believing cosine decay changes the network's architecture or weights directly
Why it failsCosine decay changes the size of update steps, not the architecture or the weights themselves.
Do this insteadDescribe the schedule as controlling how far each update moves the weights, nothing more.
Common mistake Assuming a near-zero learning rate means the model stops learning
Why it failsNear the end, the weights change only slightly from one epoch to the next. Slight change is not the same as no change.
Do this insteadDescribe late-stage updates as gentle rather than absent.
| Schedule | How the learning rate behaves | Why it matters near the loss minimum |
|---|---|---|
| Constant learning rate | Stays fixed for the entire run | One value must serve both early progress and late stability, which is a compromise |
| Cosine decay | Begins after warmup, is recalculated each epoch, and falls toward nearly zero | Smaller late steps reduce the risk of overshooting the loss minimum |
Comparing a constant learning rate with cosine decay
Reading a Schedule in a Training Configuration
When you read a training configuration that uses cosine decay, look for two things: the warmup stage that runs first, and the point where the cosine curve takes over. Together they tell you where the learning rate is high, where it is falling fastest, and where it is nearly zero.
Connect the schedule to the behavior you expect. A large early learning rate supports progress, and a shrinking learning rate in the later phases limits how far each update can carry the weights past the minimum. Thinking through this pairing makes the configuration easier to interpret.
Practice: Reasoning About a Decay Phase
A training run warms up its learning rate to a meaningful value, then switches to cosine decay. Explain, in your own words, where the decay begins, which part of the decay drops the learning rate fastest, and why the weights are less likely to overshoot the loss minimum near the end of training.
Hints
- Start by identifying the stage that runs before cosine decay.
- Think about the middle of the decay, where the learning rate is falling at its quickest pace.
- Connect a smaller step size to how far a single update can carry the weights past the bottom of the loss curve.
Key Takeaways
- Cosine decay modulates the learning rate throughout training so that it follows a cosine curve after the warmup stage, rather than staying fixed for the whole run.
- The decay begins only after warmup finishes, forming a second phase on top of the earlier one, and it ends with the learning rate reduced to nearly zero.
- The learning rate falls fastest through the middle of the decay, so the size of each update steadily shrinks, with gentle updates in the late stage.
- A smaller step reduces how far a single update can carry the weights past the loss minimum, which supports stability late in training. Cosine decay changes step size, not architecture or weights.