Learning Rate Warmup
The learning rate in cosine decay is reduced to nearly zero by the end of the schedule, which makes late-stage weight updates very gentle.
A Learning Rate That Cannot Stay Still
When a deep neural network or a large language model is trained, it repeatedly nudges its weights in a direction that should lower its loss. The loss is the measure the training process tries to reduce. The size of each nudge is controlled by the learning rate, so the learning rate is one of the most important settings in a training run.
Cosine decay is a technique that modulates the learning rate throughout training, making it follow a cosine curve after the warmup stage. In this article you will trace how that learning rate changes from one stage of training to the next, and you will see why the shape of that change matters for stability.
The Tension Behind a Fixed Step Size
A learning rate that stays fixed for the entire run is a single compromise. It must be large enough to make real progress early in training, yet small enough to avoid causing trouble later. A value that suits the beginning of training can be too aggressive near the end, and a value that is safe near the end can make the early stages painfully slow.
Cosine decay resolves this tension by letting the learning rate change over time instead of holding it constant. The learning rate is not a fixed number chosen once at the start. It is recalculated as training moves forward, so each epoch can run with a different step size than the one before it.
Where Cosine Decay Fits in a Training Run
The key phrase is after the warmup stage. Cosine decay does not govern the whole run. Training first passes through a warmup stage, and only once that stage has finished does the learning rate begin tracing the cosine curve. Decay is therefore a second phase layered on top of an earlier one, not a replacement for it.
Reading the Three Phases of the Decay
What do you think happens?
During the decay phase, in which stage does the learning rate drop at its quickest pace?
Reveal answer
In the middle of the decay
Through the middle, the learning rate is dropping at its quickest pace, so each epoch takes noticeably smaller steps than the last. At the beginning it is still close to where warmup left it, and near the end it has already settled toward a value that is nearly zero.
At the beginning of the decay phase, the learning rate is still close to where warmup left it, so updates remain fairly large. Through the middle, the learning rate drops at its quickest pace, and each epoch takes noticeably smaller steps than the last. Near the end, the learning rate has settled toward a value that is nearly zero, so the weights change only slightly from one epoch to the next.
The learning rate in cosine decay is reduced to nearly zero by the end of the schedule, which makes late-stage weight updates very gentle.
How a Shrinking Step Protects the Minimum
The loss minimum is the lowest point of the loss. When each weight update is large, the model can move so far in one step that it passes over the lowest point and lands on the other side, where the loss is higher again. This is called overshooting the loss minimum.
Cosine decay addresses this by reducing how far each update can move the weights. The gradual decrease decelerates the pace at which the model updates its weights. A slower pace leaves less room for a single step to carry the model past the minimum it is trying to settle into.
The source ties this to stability. Because the risk of overshooting is reduced, training stays steadier during its later phases. This matters most when the model is already close to a good solution, since large jumps at that point would be the most damaging.
Worked Walkthrough of a Late-Training Step
Why the same nudge behaves differently late in training
Suppose the model's weights are already close to a good solution, and the training process applies a weight update in the direction that lowers the loss. Compare what happens when the learning rate is still near its warmup level with what happens when it has decayed toward nearly zero.
- 1Identify the direction
The update points in the direction that should lower the loss, so the direction is already correct. The open question is how far the update moves the weights.
- 2Apply the step with a large learning rate
A learning rate near its warmup level sets a large step size. Near the minimum, a step this large can carry the weights past the lowest point and onto the other side, where the loss is higher again.
- 3Apply the step with a decayed learning rate
A learning rate that has decayed toward nearly zero shrinks the same update. The weights move only slightly, which leaves less room to overshoot the minimum.
- 4Compare the outcomes
The direction was identical in both cases. Only the step size changed, and that single change decides whether the model overshoots or settles near the minimum.
Cosine decay does not change which direction the weights move. It changes how far they move, and in later phases a smaller step is what protects the model from overshooting.
Common Misunderstandings About Cosine Decay
Common mistake Cosine decay changes the weights or the network's architecture.
Why it failsThe source is explicit that cosine decay changes the size of update steps, not the network's architecture or weights.
Do this insteadCosine decay acts on the learning rate, which sets how large each update step is. The weights are still updated by training itself.
Common mistake Cosine decay governs the whole training run from the first step.
Why it failsThe decay begins only after the warmup stage has finished. Warmup comes first.
Do this insteadThink of cosine decay as a second phase layered on top of warmup, not as a schedule that controls every step of the run.
Common mistake The learning rate is a fixed number chosen once at the start.
Why it failsUnder cosine decay the learning rate is recalculated as training moves forward.
Do this insteadEach epoch can run with a different step size than the one before it, so the learning rate must be understood as a changing value.
Common mistake The learning rate ends at a substantial value, so late training is just as aggressive as early training.
Why it failsThe learning rate is reduced to nearly zero, so late-stage updates become very gentle.
Do this insteadExpect the final phase of the schedule to make only small weight changes from one epoch to the next.
Common mistake Cosine decay restarts automatically each time the curve finishes.
Why it failsThe source pack describes a single descent that begins after warmup and settles toward nearly zero. It does not describe any restart behavior.
Do this insteadDo not assume a restart. Check the training configuration if you need to know whether the schedule repeats.
Applying the Schedule to a New Run
A training run uses a warmup stage followed by cosine decay. Describe the learning rate at three moments: just after warmup finishes, in the middle of the decay, and near the end of training. For each moment, say whether the weights change a lot or only a little from one epoch to the next, and explain why the loss minimum is less likely to be overshot late in training.
Hints
- Start by identifying which phase comes first and where the decay begins.
- Recall that the middle of the decay is where the learning rate drops fastest.
- Connect the size of the step to the risk of carrying the model past the lowest point of the loss.
Key Takeaways
- A fixed learning rate forces a compromise between progress early in training and stability late in training, and cosine decay resolves this by letting the learning rate change over time.
- Cosine decay begins after the warmup stage finishes and layers a second phase on top of warmup, rather than governing the whole run.
- The learning rate passes from a value near its warmup level, through a middle stretch where it drops fastest, to a value nearly zero by the end of training.
- A shrinking learning rate reduces how far each update can move the weights, which lowers the risk of overshooting the loss minimum and keeps later training stable.
- Cosine decay changes the size of update steps, not the network's architecture or weights, and the source does not describe it restarting by default.