Linear Warmup for Training Stability
A learning rate schedule can be checked by recording the rate at every training step and plotting those values.
Predicting the Shape Before Reading On
Training code that changes a learning rate can look correct on the page and still behave differently when it runs. One way to check is to record the learning rate at every training step and then draw those recorded values as a graph. The graph turns a list of numbers into a shape you can inspect at a glance.
What do you think happens?
A schedule uses a linear warmup followed by cosine decay. Before reading on, what do you think the learning rate plot looks like from the first step to the last?
Reveal answer
It rises steadily, then falls gradually
The warmup phase increases the rate until it reaches its maximum, and cosine decay then reduces it gradually until it reaches its minimum. The overall shape is a rise followed by a fall.
If your guess matched the rise-then-fall pattern, you already have the main structure. The rest of this article explains each half of that shape and shows how to tell from a plot whether the code produced it.
Recording Values to Check the Schedule
A learning rate schedule is a rule that sets the learning rate as training progresses, rather than keeping it fixed. Because the rule is applied step by step, the only reliable way to confirm what it did is to look at the values it actually produced. The method described in the source is to record the learning rate at every training step and then plot those recorded values.
The plotting call is the object of study. It takes the step indices, one for each training step, and the list of learning rates recorded during training, and draws them with labeled axes. Its purpose is to confirm that the learning rate changed as intended. So the question to keep in mind while reading the plot is what shape the intended schedule should produce. Each point on the plot should be traceable back to one step and the rate recorded for that step.
The Two Phases in Order
The schedule described here has two named phases that occur in a fixed order. The first is the linear warmup phase, and the second is cosine decay. The learning rate begins with the warmup, and cosine decay starts only after the warmup has finished. Each phase has a clear start and end, and the transition between them is marked by the rate reaching its maximum.
Read left to right, the plot should first climb in a straight-line fashion, peak at the maximum, and then slope downward. The rise is the warmup; the fall is the decay. The downward part is described as gradual, so a plot that drops sharply in the decay region does not match the schedule.
The sequence is easier to hold in mind if you treat each phase as a separate segment of the plot with its own job.
Locating the Handoff Point
Reading a plot becomes more reliable when you can say which phase any given step belongs to. Start by counting the warmup length: the source describes 20 steps of increase before the maximum. Any step inside that span belongs to warmup, and steps after it belong to cosine decay.
Assigning a Step to a Phase
Assume the warmup phase lasts 20 steps, as the source describes. Which phase does step 12 belong to, and which phase does step 40 belong to? This question is generated for practice and is not a value taken from the source.
- 1Identify the warmup span
The warmup phase lasts 20 steps, so the ramp covers the early part of the run.
- 2Check step 12
Step 12 falls inside the 20-step warmup span, so the learning rate at that step should still be on the rising segment.
- 3Check step 40
Step 40 is well past the end of the warmup span, so it belongs to cosine decay and the rate should be on the falling segment.
- 4Note the boundary
Whether the handoff lands exactly at step 20 or step 21 depends on how the code counts steps. Check the counting convention in the code before judging a single point at the boundary.
Step 12 is in the warmup phase and step 40 is in the cosine decay phase.
Finding Where a Plot Departs
When a learning rate plot does not match what you expected, the useful question is where it first diverges. Because the schedule has two ordered phases, check them in sequence. First confirm that the early segment rises steadily for the warmup length. Then find the point where the rise stops and check that the decline after it is gradual.
| Symptom on the plot | Phase where the problem lies | Why |
|---|---|---|
| Curve falls before the warmup has finished | Warmup phase | The rate should still be rising during the warmup length |
| Curve keeps rising after the warmup should have ended | Handoff to decay | Cosine decay should have taken over from the peak |
| Curve drops sharply during decay | Decay phase | The source describes a gradual reduction, not a sharp drop |
Mapping plot symptoms to the phase that should be checked first
Mistakes When Reading a Schedule Plot
Common mistake Judging the whole curve at once and calling it correct or wrong
Why it failsA plot can look roughly right overall while one phase is wrong, so the overall impression hides the problem.
Do this insteadWalk through the phases in order and find the first step where the curve stops matching the expected phase.
Common mistake Assuming the plot shows the code's intent instead of its recorded values
Why it failsThe plot is only as accurate as the values recorded at each step.
Do this insteadCheck that each plotted point comes from a recorded learning rate paired with its step index.
Common mistake Treating a single point at the boundary as proof of a bug
Why it failsWhether the handoff falls at step 20 or 21 depends on the step counting convention used in the code.
Do this insteadConfirm the counting convention first, then judge whether the sustained shape changes at the expected place.
Common mistake Expecting the decay phase to drop sharply rather than gradually
Checking a Plot Yourself
A recorded schedule with a 20-step linear warmup and cosine decay produces a plot that climbs steadily through the first 20 steps, then falls gradually. Suppose a second run's plot climbs steadily for 20 steps but then keeps climbing for another 10 steps before any decline. Which phase should you check first, and why? This practice question is generated and not taken from the source.
Hints
- Compare the observed curve with the expected handoff point at the end of the warmup.
- The source's rule is to find the first step where the curve stops matching the expected phase.
- The symptom of rising past the warmup end points to the handoff.
Key Takeaways
- A linear warmup raises the learning rate for 20 steps until it reaches its maximum, and cosine decay then reduces it gradually until it reaches its minimum.
- Recording the learning rate at every step and plotting the values turns the schedule into a shape you can inspect.
- To debug a plot, find the first step where the curve stops matching the expected phase rather than judging the whole shape at once.
- A fall during warmup, a continued rise after warmup, or a sharp drop during decay each point to a different phase to check.