Loss Functions for Binary Classification

This article begins with a script rather than a definition. The script builds a tiny model with one input, one weight, and one bias. It passes the input through a sigmoid and then through a binary cross-entropy loss, and finally asks PyTorch for the gradient of that loss with respect to a parameter. As you trace it, watch how each variable receives its value and what the final gradient looks like when it is printed.

Keep one question in mind as you read: at which line does the state of the program stop matching what you would expect? Manual gradient code is short, so a mistake usually hides in one specific line. Finding that line is much faster when you already know what each line is supposed to produce.

Before any gradient is requested, the script has already built a chain of operations. The input x1 is multiplied by the weight w1, the bias b is added to produce z, the sigmoid of z gives a, and the binary cross-entropy of a against the target y gives the loss. PyTorch records each of these steps as it runs. That recorded chain of operations is what people mean by the computation graph.

The diagram shows the direction of data during the forward pass, from inputs toward the loss. Gradients run the other way, from the loss back toward the parameters. The parameters w1 and b are marked as requiring a gradient because the script creates them with requires_grad set to True. Tensors without that flag, such as the input x1 and the target y, are not the variables whose gradients this script asks for.

The grad function is imported from torch.autograd. It takes the loss and the variable you want to differentiate with respect to. In the script, the call grad(loss, b, retain_graph=True) asks how the loss changes as the bias b changes. The result is stored in a variable named grad_L_b. The same pattern works for any parameter created with requires_grad set to True, such as the weight w1.

The gradient answers a precise question: how does the loss change when one parameter changes? Written with calculus notation, the gradient of the loss L with respect to the bias b is $\frac{\partial L}{\partial b}$. The grad call is how you ask Autograd to compute this quantity for a tensor it has been tracking.

By default, PyTorch destroys the computation graph after calculating the gradients, in order to free memory. This is a sensible default. A graph can hold many intermediate values, and once the gradient has been computed, a standard training step usually does not need them again. The consequence is that the graph is no longer available for another gradient request unless you ask PyTorch to keep it.

The script changes that default with retain_graph=True. This keyword argument tells PyTorch to keep the graph in memory after the call finishes. The reason is stated directly in the source: the script will reuse the same computation graph shortly, so the graph must stay available. Without that setting, a second request against the same loss can fail because the graph it needs has been released.

The source names two gradients that come from the same loss: one with respect to the weight w1 and one with respect to the bias b. The shared loss is the reason the graph must survive the first request. The code below is an original illustration of the same ordering pattern, using the same kind of model so the mechanism stays concrete.

When a manual gradient script gives a result you did not expect, work through it in order rather than guessing. Start with the setup and confirm which tensors were created with requires_grad set to True and that the loss was built from them. Next, check each grad call: confirm which variable is being differentiated and whether retain_graph=True is present on every call that needs the shared graph. Finally, check the printed output and remember that the values come back wrapped in a tuple.