Linear Warmup in LLM Training

Training a large language model means repeatedly nudging its parameters so that its predictions improve. Each nudge is driven by gradients, and most of the time those gradients are well behaved. Occasionally, however, a gradient becomes far larger than the rest, and a single update built from it can push the parameters much further than intended. The result is an unstable training run.

This article focuses on one remedy, gradient clipping, and on what it does to the parameters that training is trying to adjust. Clipping is a technique used to prevent exploding gradients by limiting the gradient norm to a maximum value.

A gradient tells training how each parameter should change to reduce the error. It has a direction, which says whether a parameter should go up or down, and a magnitude, which says how strongly. The overall size of the gradient is often described by its norm, a single number that summarizes how large the gradient is taken as a whole. Written compactly, the norm of a gradient vector $g$ is $\|g\|$.

The key idea behind clipping is to put a ceiling on that overall size. Call the ceiling the maximum norm. If the gradient is already within the ceiling, nothing is done to it. If it exceeds the ceiling, it is scaled back so that its size no longer goes past the maximum. Clipping therefore acts on the size of the whole gradient, not on each parameter separately.

Gradient clipping is not used alone in the improved version of the training loop. The source describes an updated training function that takes the earlier train_model_simple function and adds three concepts: linear warmup, cosine decay, and gradient clipping. Clipping is the piece that directly limits the size of gradients, while linear warmup and cosine decay are the other two additions introduced alongside it.

The flow below traces one iteration. The backward pass produces gradients, clipping limits their overall size, and the optimizer then uses the clipped gradients to update the parameters. The learning-rate schedule, which is where warmup and cosine decay live, also feeds into the update.

The clearest way to see clipping is to follow one gradient through it. Suppose the gradient has two components, 3 and 4, and the maximum norm is 1. The norm of this gradient is 5, which is well above the ceiling, so clipping applies. The example below uses illustrative numbers that are not from the source; they are chosen so the arithmetic is easy to check.

Clipping scales the whole gradient by the same factor, here 1/5, so that the new norm equals the maximum of 1. Both components shrink, and the ratio between them stays the same: 3 and 4 become 0.6 and 0.8. The largest entry went from 4 to 0.8, which is substantially smaller, matching what the source describes for the largest gradient value after clipping with a maximum norm of 1.