Computing Gradients via Autograd

Gradient computation is easiest to understand when you have a guess about the answer before the library hands you a number. This article follows one short PyTorch program. The program takes an input, multiplies it by a weight, adds a bias, passes the result through a sigmoid, and measures the error with a binary cross-entropy loss. PyTorch's autograd system then produces gradients for the parameters. The purpose of the program is to show two different ways of asking autograd for those gradients, and how the order of the calls affects what you can read back.

Start with the setup. The bias b begins at 0.0 and is tracked by autograd. The input x1 is 1.1, the weight w1 is 2.2, and the target y is 1.0. Before reading further, decide whether the gradient of the loss with respect to b should be positive or negative. The target is 1.0 and the sigmoid output for this starting point is below 1, so the prediction is too low. Commit to a sign before you read the answer.

Every value in the program is a link in a chain. The weighted sum z is built from w1, x1 and b. The sigmoid output a is built from z. The loss is built from a and the target y. Autograd records these links as a computation graph while the forward values are computed, so that a later request for a gradient can walk back along the same links from the loss toward the parameters.

The tree shows the path a backward pass follows. The loss depends on a, a depends on z, and z depends on w1, x1 and b. The leaf tensors are the ones at the bottom of the tree that autograd can deliver a gradient to. Because x1 was created without requires_grad, it is on the path but does not collect a gradient of its own.

The code below reconstructs the program described above. It has two phases. The first phase builds the loss and calls grad once to get the gradient of the loss with respect to b. The second phase calls backward on the same loss and reads the grad attributes of the parameters.

Notice that the code never passes w1 or b to backward. The loss already knows which tensors fed into it through the recorded graph. The grad call is different: it answers one question about one chosen tensor and returns the answer to you. The backward call answers the question for every tracked tensor at once and stores each answer in that tensor's grad attribute.

The grad function computes one gradient on request and returns it. It takes the loss and the parameter whose gradient you want. The backward method is called on the loss tensor and computes gradients for every tracked tensor that contributed to it, storing each result in that tensor's grad attribute. The two differ in where the answer goes and how many answers come back.

Both phases of the program depend on the same computation graph. The grad call uses the graph to produce grad_L_b. The argument retain_graph=True tells autograd to keep that graph so the later loss.backward() call can still use it to fill in w1.grad and b.grad. This is why the order of the calls and the retain_graph argument matter together.

Only tensors created with requires_grad=True receive gradients. In the program, w1 and b have that setting, so they are the tensors whose grad attributes are filled in. The tensor x1 does not, so it takes part in the forward computation but never gets a gradient. This means the requires_grad choice decides what you can read back after the calls.

When a printed gradient looks wrong, do not start with the final number. Start with the program's structure. First check which tensors have requires_grad=True, since only those receive grad attributes. Then check which call produced the value you are reading: grad returns a value, while backward fills grad attributes. Finally, trace the forward values through the same operations to find the first place where your expected value and the computed value disagree.

This hand check is tracing, not a replacement for autograd. In practice you let backward compute the value. You trace by hand only to confirm which tensor and which operation a number came from when something looks wrong.