Gradient Calculation in PyTorch

A neural network learns by adjusting its weights. Each weight receives a gradient, and that gradient tells the training process how much the weight should change and in which direction. In most steps these gradients are moderate, and training proceeds smoothly. Sometimes, however, a gradient becomes very large. When that happens, a single update can move the weights far away from where they were, and training can become unstable. This problem is known as exploding gradients.

Gradient clipping is the technique used to prevent exploding gradients by limiting how large the gradients can become. In this article you will follow a short sequence of PyTorch calls, watch a single value at several points, and work out what each step contributed. The goal is not only to remember the function name, but to be able to tell exactly where in the sequence a result stops matching your expectations.

A training iteration has a fixed order. The forward pass runs the model on a batch of inputs and produces predictions. A loss function compares those predictions with the targets. The loss.backward() call then computes gradients. Gradient clipping comes after .backward(), and the optimizer step that uses the gradients to update weights comes last. The order is not a formality: each stage depends on the output of the one before it, so the order of calls determines what each printed result actually means.

Clipping sits between backward and the optimizer step because it needs gradients to exist (they are created by backward) and because the update should use the already limited values. Placing it before backward would mean there is nothing to limit. Placing it after the optimizer step would mean the weights were already updated with the unlimited gradients.

The gradients are not returned by backward() and they are not stored in a separate gradient table. Instead, PyTorch attaches each one to the parameter it belongs to. After the backward call, each weight tensor that took part in the computation has a .grad attribute holding its gradient. This is why any code that reads gradients must run after .backward(). Before the backward call, the .grad attributes do not yet contain values from the current batch.

The diagram shows the idea in simplified form. The backward pass starts from the scalar loss and reaches each weight tensor that the loss depends on. Each weight then holds its own gradient in its .grad attribute. A learner who wants to inspect the result can read the .grad field of a specific parameter, which is exactly what the utility in the next section does across all parameters at once.

The source uses a utility named find_highest_gradient(model) to report one number: the largest gradient value across the model. It scans every parameter that has a .grad value, flattens each gradient tensor, and keeps the largest value it finds. Because it reads .grad, it gives a meaningful answer only after .backward() has run. The code below is an illustrative version written for this article, not the source's exact implementation. It uses the magnitude of each gradient so that the largest value is compared regardless of sign.

Gradient clipping limits the size of gradients to help prevent exploding gradients. In PyTorch it is applied with torch.nn.utils.clip_grad_norm_(). In this course the call uses max_norm=1.0. The call takes the model's parameters, and it works on their gradients. Its documented behavior is to measure the norm of the gradients together and, when that combined norm exceeds max_norm, to scale the gradients down so the norm equals max_norm. When the combined norm is already at or below max_norm, the gradients are left as they are.

The before and after panels describe the same parameters at two moments. When clipping triggers, every gradient is scaled by the same factor, so the largest value shrinks along with the rest. The utility is the way to observe this effect, since it reports the highest value at each point in the sequence.

A common question is whether the threshold is applied to each parameter separately or to one combined value. The clipping call measures the gradients together across the parameters it receives, so the limit is a single shared threshold rather than a separate cap on every weight. A single parameter with an unusually large gradient can therefore lower the scale applied to all the gradients.