Tensors and requires_grad in PyTorch

A PyTorch program can build a loss from an input, a weight, a bias, a sigmoid, and a binary cross-entropy calculation. Once that loss exists, you can ask for a gradient in two different ways: torch.autograd.grad returns a requested gradient, while loss.backward() computes gradients for tracked tensors that contributed to the loss and stores them in their grad attributes. The distinction is not just what you ask PyTorch to calculate; it is also where you get the result.

Follow the same small computation throughout this article. Its input is x1 = 1.1, its weight is w1 = 2.2, its bias is b = 0.0, and its target is y = 1.0. The forward pass multiplies the input by the weight, adds the bias, applies sigmoid, and measures the error with binary cross-entropy. The key question is how the gradient request and the tracked tensors determine what you can read afterward.

The loss is the end of a chain of operations, not an isolated number. The input and weight feed a multiplication; that result and the bias feed an addition; the result passes through sigmoid; and the sigmoid output is compared with the target by the binary cross-entropy loss. Autograd records the tensor operations in this forward pass so it can trace gradients back from the loss.

To investigate a surprising gradient, trace this chain rather than staring only at the final value. Start with the loss and identify the operations that produced it; then follow their inputs backward toward the weight, bias, and input. The first link where your expected value and the program's value disagree is the place to inspect.

The requires_grad setting determines which tensors are tracked for gradients. The source example tracks the bias b and reads gradients for both w1 and b after backward. More generally, only tensors created with requires_grad=True receive gradients. A tensor also needs to contribute to the loss through the recorded computation for that loss to provide a path back to it. If a parameter has requires_grad=False, do not expect a gradient for it from this tracked path.

torch.autograd.grad asks for a gradient with respect to a chosen tensor and returns that gradient as the function result. In the example, grad(loss, b, retain_graph=True) asks for the loss gradient with respect to b, and the returned value is assigned to grad_L_b. By contrast, loss.backward() does not name a parameter to request; the loss's recorded graph identifies the tracked tensors that contributed to it, and their gradients are stored in their grad attributes.

Read the result from the place that matches the call. The value returned by grad is the answer to its one requested gradient question; backward fills the grad attributes of tracked tensors on the path to the loss. In the example, grad_L_b comes from the first call, while w1.grad and b.grad are read after the second.

Both gradient calls in the example rely on the same computation graph. The program calls grad first and then calls backward on the same loss. Passing retain_graph=True to grad preserves the graph for the later backward call, allowing backward to fill w1.grad and b.grad after grad has already returned grad_L_b.

The source setup also invites a sign prediction for the bias gradient: b starts at 0.0, the target is 1.0, and the sigmoid output is below 1. Commit to a sign before checking the program's result. That prediction is a hand check of the traced computation, not a substitute for letting autograd calculate the gradient.