Learning Rate Schedules

Training code that changes a learning rate can look correct on the page and still behave differently when it runs. One practical way to check is to record the learning rate at every training step and then draw those recorded values as a graph. The graph turns a list of numbers into a shape you can inspect at a glance. Before reading any explanation, try to guess what that shape should be.

The schedule described in this article has two named phases that occur in order. The first is the linear warmup phase, and the second is cosine decay. The learning rate starts with the warmup, and cosine decay takes over only after the warmup has finished. Because the phases are ordered, you can describe any step in a run by asking which phase it belongs to.

During the warmup phase the learning rate increases in a linear way. In the schedule this article describes, the warmup lasts 20 steps and the rate climbs until it reaches its maximum. The index map below marks the three positions that matter most when you trace the early part of the curve: the first step, the end of warmup, and the later steps where decay happens.

Cosine decay starts only after the warmup ends. At that handoff the rate has reached its maximum, and from there it is reduced gradually until it reaches its minimum. The timeline below shows the order of these transitions, with the peak marking the boundary between the two phases.

The check works in three connected parts. The scheduler sets the learning rate for each step. The training run records the rate it used at every step. A plotting call then takes the step indices, one per training step, together with the recorded learning rates, and draws them with labeled axes. The plot is only as trustworthy as the recording, so the question to keep in mind is what shape the intended schedule should produce.

The purpose of the plotting call is to confirm that the learning rate changed as intended. A plot that matches the rise-then-fall pattern is evidence that the schedule is doing what it was written to do. The plot turns a list of numbers into a shape, and that shape is what you compare against the expectation.

When a plot does not match what you expected, the useful question is where it first diverges. Because the schedule has two ordered phases, you can check them in sequence. First confirm that the early segment rises steadily for the warmup length. Then find the point where the rise stops and check that the decline after it is gradual.

A warmup-then-cosine schedule rises linearly for 20 steps to its maximum, then falls gradually toward its minimum under cosine decay. Recording the rate at every step and plotting those values turns the schedule into a shape you can check. When a plot disagrees with that shape, locate the first step where it departs, rather than judging the whole curve at once.