Leaf Nodes and the Computation Graph

Up to this point, gradients have been computed by calling the grad function directly. That approach is useful for experimenting, for debugging, and for showing the underlying ideas. In practice, though, PyTorch offers a higher-level tool that does the work for you. This article starts with the code that uses that tool and then explains what happens when the code runs.

Imagine a loss value that has already been computed from a model's parameters, where the parameters w1 and b are tensors. A short program calls backward on the loss and then prints the gradients that PyTorch stored for two of those parameters. Notice that the program never writes a derivative formula. The only calculus step in it is the single call to backward. The calculus is still happening underneath; you simply no longer write it out yourself.

The forward pass builds a computation graph: the loss was produced by a sequence of operations, and each operation links its inputs to its output. Calling backward on the loss starts at the loss and traverses that graph in reverse. The traversal reaches the leaf nodes, which are the tensors the graph was built from, such as the parameters w1 and b. Gradients are computed for all of these leaf nodes at once.

Read the diagram from top to bottom. The path starts at the loss, and the backward call moves through the graph until it reaches the leaf nodes. Each leaf node receives its gradient, and that gradient becomes available through the matching .grad attribute. The parameters w1 and b are the places where the values finally land. You do not have to follow the chain of calculus steps yourself; the traversal handles it.

The distinction matters because the important detail is which tensors receive gradients. Calling backward on the loss computes gradients for the leaf nodes. The intermediate results sit between the leaves and the loss, and the gradients you read come from the leaves. This is why you read gradients from parameters like w1 and b and not from every value that appeared during the computation.

Each leaf tensor has its own .grad attribute, and that is where its gradient is stored. The gradient for a given parameter is looked up directly from that parameter. Before backward runs, the attribute holds no computed gradient. After backward finishes, the attribute holds the gradient that PyTorch computed for that leaf.

The source output shows one gradient value for w1 and one for b. Those are the values you read from the two .grad attributes after the backward call. Reading them is the only step you perform; the computation was done when backward ran.

The source moves from gradients to a concrete network that PyTorch is often used to build: the multilayer perceptron. A multilayer perceptron is a fully connected neural network with multiple hidden layers between its input and output layers. Fully connected means that the units in one layer connect to the units in the next layer.

The source illustrates this structure with two hidden layers. Each node in its drawing stands for one unit in its layer, and the drawing uses very few nodes per layer so the structure is easy to see. The input layer is where data enters, the hidden layers sit between input and output, and the output layer is where the network produces its result. The source does not specify exactly which layer each named parameter such as w1 or b belongs to, so this article does not assign them to specific layers.

When a gradient is missing or unexpected, the source suggests checking three things: the call order, whether the tensor is a leaf node, and which loss was differentiated.