Manual Gradient Computation with the grad Function
Calling .backward on the loss computes gradients for the leaf nodes of the graph, so you do not need to derive them by hand.
Starting From a Working Script
Training a neural network means adjusting its parameters so that a loss value gets smaller. To make those adjustments, you need to know how the loss changes when each parameter changes. That quantity is the gradient of the loss with respect to a parameter. Mathematically it is a derivative, and for a model with many parameters, computing those derivatives by hand would be slow and error-prone.
PyTorch offers a higher-level route. You compute a loss from your parameters, call backward on that loss, and then read the gradients PyTorch stored for you. The calculus still happens underneath, but your code does not write out any derivative formula. This article starts with a small script that uses this workflow, then explains what happens when that script runs. Along the way you will also meet the lower-level grad function, which asks for a single gradient directly and is useful for experiments and debugging.
What do you think happens?
After loss.backward() runs on a loss built from a parameter w1 created with requires_grad=True, what does w1.grad contain?
Reveal answer
The gradient of the loss with respect to w1
Calling backward on the loss computes gradients for the leaf nodes of the graph, and each leaf tensor's gradient is stored in its .grad attribute. The parameter w1 is a leaf node, so its gradient lands in w1.grad.
Keep one question in mind as you read the trace in this article: at which line does the state of the program stop matching what you would expect? Manual gradient code is short, so a mistake usually hides in one specific line. Finding that line is much faster when you already know what each line is supposed to produce.
Recording the Forward Pass
Before any gradient is requested, the script has already built a chain of operations. The input x1 is multiplied by the weight w1, the bias b is added to produce z, the sigmoid of z gives a, and the binary cross-entropy of a against the target y gives the loss. Each of these steps is recorded by PyTorch as it runs. That recorded chain of operations is what people mean by the computation graph.
- x1 and y are plain tensors; they do not ask PyTorch for gradients
- w1 and b are created with
requires_grad=True, so Autograd tracks them - z, a, and loss are intermediate values produced by operations as the script runs
Two terms from this script matter for everything that follows. A leaf node is a tensor that the graph was built from, such as a parameter like w1 or b, rather than a value produced by an intermediate operation. The values z and a are produced by operations, so they are not leaf nodes. The parameters are the tensors whose gradients this workflow is designed to produce.
Tracing the Backward Pass
The direction matters. The forward pass runs from the inputs toward the loss. Gradients run the other way, from the loss back toward the parameters. When you call backward on the loss, PyTorch traverses the recorded graph in reverse order, starting at the loss and moving back through each operation until it reaches the leaf nodes. Each leaf node receives its gradient, and that gradient is then available through the matching .grad attribute.
Read the diagram from top to bottom. Start at the loss, follow the arrows back through the sigmoid and the weighted sum, and notice that the parameters w1 and b are the places where the values finally land. Nothing in the diagram is a formula you have to solve. The only calculus step in the program is the single call to backward.
The calculus is still happening underneath. The earlier discussion of gradients explained the math, and that math is what backward relies on. What changes is your workflow. You do not compute derivatives by hand, and you do not need to write out the chain of calculus steps to train with them.
Where Gradients Are Stored
The computed gradients are stored in each leaf tensor's .grad attribute, and that is where you read them. Before backward is called, a parameter's .grad attribute has no gradient yet. After backward runs, the leaf tensors that the loss depends on hold their gradients in .grad. Intermediate values such as z and a are not leaf nodes, so the gradient results you read are the ones on the parameters.
loss.backward() # not yet called
print(w1.grad) # no gradient stored yetw1.grad and b.grad have no gradient stored (None)
loss.backward()
print(w1.grad)
print(b.grad)w1.grad and b.grad each hold the gradient of the loss with respect to that parameter
The code that reads these results is short. The first line calls backward on the loss, and the two print lines then read the gradients that PyTorch stored for the two parameters. The .grad attribute is where those values are read, so you access it with a dot rather than calling a function.
loss.backward()traverses the graph and fills in gradients on the leaf tensors- w1.grad reads the gradient stored on the parameter w1
- b.grad reads the gradient stored on the parameter b
Asking grad() for One Gradient
The grad function is imported from torch.autograd. It takes the loss and the variable you want to differentiate with respect to. Calling grad(loss, b) asks how the loss changes as the bias b changes. The same pattern works for any parameter created with requires_grad set to True, such as the weight w1. Unlike backward, grad returns the gradient to you directly instead of storing it on the leaf tensor's .grad attribute.
- grad is imported from torch.autograd, not called as a method on the loss
- loss is the first argument, the value being differentiated
- b is the second argument, the variable whose gradient is requested
retain_graph=True tells PyTorch to keep the graph for later use- the result is stored in the variable grad_L_b
Notice the shape of what grad returns. The gradient is wrapped in a tuple, so the printed form includes a parenthesis and a trailing comma. The tuple wrapping is part of what you see in the trace, and it is the first thing to check when a printed gradient looks unfamiliar. If you need the single tensor, you index into the tuple before using it.
Tracing a two-gradient script
Given a loss built from w1 and b, request the gradient with respect to w1 and then with respect to b, both from the same loss. Explain what each line does and why the first call must keep the graph.
- 1Build the graph
The forward pass records x1 times w1, the addition of b, the sigmoid, and the loss. The leaf nodes are w1 and b because both were created with
requires_grad=True. - 2First grad call
grad(loss, w1, retain_graph=True)asks for the gradient with respect to w1. Theretain_graph=True setting keeps the graph in memory after this call finishes. - 3Second grad call
grad(loss, b, retain_graph=True)asks for the gradient with respect to b from the same loss. This works only because the graph survived the first call. - 4Check the result
Each result is a tuple. Read the value from the tuple before using it, and confirm that the variable named in each call is the one you intended to differentiate.
Both gradients come from one shared graph. The first call's retain_graph=True is the reason the second call can succeed.
Multilayer Perceptrons as Layered Graphs
The source moves from gradients to a concrete kind of network that PyTorch is often used to build: the multilayer perceptron. It is a fully connected neural network, meaning the units in one layer connect to the units in the next. A multilayer perceptron has an input layer where data enters, one or more hidden layers between the input and output, and an output layer that produces the result. The source illustrates one with two hidden layers.
Each node in the source drawing stands for one unit in its layer, and the drawing uses very few nodes per layer so the structure is easy to see. Real networks have many more units. The layered picture matters for gradients because the loss is computed at the output, and the gradient must travel back through every layer to reach the parameters that sit between them.
A multilayer perceptron used for classification passes its input through hidden layers and then produces an output that is compared with a target through a loss. Calling backward on that loss is what sends gradients back through the output layer and the hidden layers to reach the parameters, which is why the same .grad workflow applies to networks with many layers.
Why the Graph Is Freed by Default
By default, PyTorch destroys the computation graph after calculating the gradients, in order to free memory. This is a sensible default. A graph can hold many intermediate values, and once the gradient has been computed, a standard training step usually does not need them again. The consequence is that the graph is no longer available for another gradient request unless you ask PyTorch to keep it.
The script changes that default with retain_graph=True. This keyword argument tells PyTorch to keep the graph in memory after the call finishes. The reason is direct: the script reuses the same computation graph shortly, so the graph must stay available. Without that setting, a second request against the same loss can fail because the graph it needs has already been released.
Mistakes That Break Manual Gradient Scripts
Common mistake Requesting a second gradient from the same loss without retain_graph=True
What it looks likegrad(loss, w1) followed by grad(loss, b), with no retain_graph setting on the first call
Why it failsBy default PyTorch frees the graph after the first gradient is computed, so the second request has no graph left to use.
Do this insteadAdd retain_graph=True to the calls that need the shared graph.
Common mistake Creating a parameter without requires_grad=True
What it looks likew1 = torch.tensor(0.8) used as a weight in the lossWhy it failsAutograd tracks only tensors created with
requires_gradset to True, so no gradient is produced for w1.Do this insteadCreate parameters with requires_grad=True before building the loss from them.
Common mistake Forgetting the tuple wrapping that grad returns
What it looks likeUsing grad_L_b directly as a number after calling grad(loss, b)
Why it failsgrad returns its results inside a tuple, so the printed form includes a parenthesis and a trailing comma.
Do this insteadIndex into the tuple to get the tensor before doing arithmetic with it.
Common mistake Looking for gradients on intermediate values such as z or a
Why it failsBackward stores gradients on leaf nodes such as parameters, not on values produced by intermediate operations.
Do this insteadRead gradients from the parameters, or request them explicitly with grad against the parameter you care about.
Common mistake Calling grad on a variable that is not part of the loss's graph
Finding Where a Result Goes Wrong
When a manual gradient script gives a result you did not expect, work through it in order rather than guessing. The source recommends checking the setup first, then each grad call, then the printed output, in that order. Most problems fall into one of those three places.
- 1Check the setup: confirm which tensors were created with
requires_gradset to True and that the loss was built from them. - 2Check each grad call: confirm which variable is being differentiated and whether
retain_graph=True is present on every call that needs the shared graph. - 3Check the order of calls: confirm the loss was built before the grad call that uses it, and that no earlier call already freed a graph you still need.
- 4Check the printed output: remember the values come back wrapped in a tuple, and read the tensor from inside it.
Practice Predicting the Graph State
Using the script from the grad section, you decide to request only the gradient with respect to w1 first, and then the gradient with respect to b. Which call needs retain_graph=True, and what goes wrong if it is omitted? Explain in one or two sentences.
Hints
- Ask which call happens first and which one needs the graph to still exist afterward.
- The graph is freed by default after gradients are calculated, so the first call is the one that must keep it.
Key Takeaways
- Calling backward on a loss computes gradients for the leaf nodes of the computation graph, so you do not derive them by hand. The gradients are stored in each leaf tensor's .grad attribute.
- Before backward is called, a parameter's .grad holds no gradient yet. After backward, the leaf tensors that the loss depends on hold their gradients.
grad(loss, tensor)from torch.autograd asks how the loss changes with respect to one chosen tensor and returns the result in a tuple. Parameters needrequires_gradset to True for Autograd to track them.- PyTorch frees the computation graph after gradients are calculated by default. Setting
retain_graph=True keeps it available for another gradient request against the same loss. - When a manual gradient result looks wrong, check the setup, then each grad call, then the printed output, in that order.