Logistic Regression as a Computation Graph
PyTorch's autograd system computes gradients in dynamic computational graphs automatically, and it records the computation that produces the loss so gradients can be found later.
Try it: Neural Network Forward Pass
How a neural network turns inputs into an output: each neuron computes a weighted sum plus bias, applies an activation function, and passes the result on.
How it works
- Each hidden neuron computes z = w1·x1 + w2·x2 + b.
- It applies an activation: ReLU keeps positive values and zeroes negatives; sigmoid squashes z into 0–1.
- The output neuron does the same with the hidden activations as its inputs.
- Changing any input, weight or bias changes every value downstream of it.
Default run (7 steps): Inputs x1 = 1, x2 = 0.5. Activation: ReLU. … Prediction y = ReLU(1.25) = max(0, 1.25) = 1.25.
Simplified: Tiny 2-2-1 network with hand-set weights, not a trained model. No training happens here. Activations offered: ReLU, sigmoid and sign (texts differ on the value of sign at exactly 0; here it is 0).
Loading the simulation…
A Prediction Built From Small Steps
A model's prediction often feels like a single action: you give it an input and it returns an answer. Underneath, the answer is produced by a chain of small operations, each one consuming the results of the one before it. This article takes a small classifier and asks a practical question: how do we write down that chain so that every intermediate value has a place, a name, and a clear route to the final result?
The answer is the computation graph. A computation graph is a sequence of computations drawn as a directed graph, so every step is a node and every arrow points in the direction that data travels. Once a calculation is drawn this way, you can point at any value and see what produced it and what it will feed into next. PyTorch uses this same idea internally, and by the end of this article you will be able to read the graph that a small logistic regression classifier leaves behind.
Why Training Needs a Recorded Trail
Training a model means adjusting its parameters so that the loss gets smaller. To adjust a parameter sensibly, you need to know how much a small change to that parameter would change the loss. That quantity is a gradient. For a model with many parameters, working out these gradients by hand for every calculation becomes impractical. This is the problem PyTorch's autograd system is built to handle.
PyTorch's autograd system is its automatic differentiation engine. It provides functions that compute gradients in dynamic computational graphs automatically. The key idea is that PyTorch does not only compute your loss. It also keeps track of how the loss was produced, so that the gradients can be worked out afterward. Before looking at that bookkeeping, ask a simple question: what would you need to know about a calculation in order to find its gradients without redoing it from scratch? The answer is a record of which values were computed from which others, and that record is the graph.
When PyTorch Starts Recording
PyTorch builds a computation graph in the background. The graph is a record of the operations that turned inputs into outputs, arranged so that each result points to the values it was calculated from. Its purpose is specific: it lets PyTorch calculate gradients of a loss function with respect to the model parameters, which is what training needs.
The trigger for building this graph is a flag on the tensors involved. If you carry out computations in PyTorch, it builds a computational graph internally by default as long as one of the terminal nodes has its requires_grad attribute set to True. The terminal nodes are the values at the end of the computation, and requires_grad marks the quantities for which you want gradients.
Once the graph exists, gradients are found by applying the chain rule from calculus. Gradients are required when training neural networks via the backpropagation algorithm, which can be considered an implementation of the chain rule for neural networks. The most common way of computing loss gradients in a computation graph applies the chain rule from right to left. This is also called reverse-mode automatic differentiation or backpropagation.
The Parts of a Logistic Regression Classifier
A logistic regression classifier is a single-layer neural network whose forward pass multiplies the input by a weight, adds a bias, and applies a sigmoid to return a score between 0 and 1.
- Input feature: the raw number describing one example. It comes from the data.
- Trainable weight: a parameter that scales the input. The model adjusts it over time.
- Trainable bias unit: a parameter that shifts the result. The model adjusts it over time.
- Sigmoid: the function that turns the net input into the output a, keeping that output between 0 and 1.
- True label: a quantity that is not used to make the prediction. It is used only at the loss, where the model output is compared with it.
The first two steps of the forward pass compute the net input z, which is the input multiplied by the weight with the bias added, and then apply the sigmoid to produce the output a. Written compactly, the net input and the output are:
Here x is the input feature, w the weight, b the bias, z the net input, and a the output. The sigmoid keeps a between 0 and 1, which is why the classifier can report a score. The score is not the end of the story. During training it is compared with the true class label, which is either 0 or 1, and that comparison produces a loss. Every piece of the forward computation exists to feed that loss.
Tracing the Forward Pass With Numbers
One forward pass with x = 2, w = 0.5, b = 0.1
Given an input feature x = 2, a weight w = 0.5, and a bias b = 0.1, find the net input z and the output a of the classifier.
- 1Compute the net input
Multiply the input by the weight and add the bias: z = 0.5 times 2 plus 0.1 = 1.0 plus 0.1 = 1.1. The net input cannot be computed until x, w, and b all exist.
- 2Apply the sigmoid
The output is a =
sigma(1.1)= 1 / (1 + e-1.1). Since e-1.1 is about 0.3329, the output is about 1 / 1.3329, which is roughly 0.750. - 3Record the intermediate values
The sequence of intermediate results is x, w, b, then z = 1.1, then a of about 0.750. Each value depends only on the values computed before it.
The net input is z = 1.1 and the score is a of about 0.750, a value between 0 and 1.
The Forward Pass as a Directed Graph
Now redraw the forward pass as a graph. Each input and parameter becomes a starting node. Each operation becomes a node that takes its inputs from earlier nodes. Arrows point from the values that are used toward the results that use them. Reading the arrows as one-way dependencies tells you which values must be ready before each step can run. The diagram below shows this form, with the true label feeding into the loss at the far end.
The graph makes three distinctions visible that the code only hints at. Input features and parameters sit at the top because they are the starting values. Intermediate results, such as the net input z, sit in the middle because they exist only to pass information along. The output and the loss sit at the bottom because they are the end products. The label enters from the side, because it never feeds the prediction, only the comparison that follows it.
Writing the Forward Pass in PyTorch
The listing below is a small illustration written for this article, not the source's own listing. It runs the same forward pass as the worked example and then asks PyTorch for gradients. Notice which tensors carry requires_grad. Only the weight and bias carry it, so only they get gradients, while the input and label do not. The squared error used for the loss is a choice made for this illustration; the source does not fix one particular loss.
import torch
x = torch.tensor(2.0)
w = torch.tensor(0.5, requires_grad=True)
b = torch.tensor(0.1, requires_grad=True)
y = torch.tensor(1.0)
z = w * x + b
a = torch.sigmoid(z)
loss = (a - y) ** 2
loss.backward()
print(w.grad, b.grad)- 1Lines 3 to 6 create the input, the two trainable parameters, and the label as tensors.
- 2Line 8 computes the net input z from the parameters and the input. PyTorch records this operation.
- 3Line 9 applies the sigmoid, so a is the score between 0 and 1.
- 4Line 10 compares a with the label. This is the only place the label enters the computation.
- 5Line 12 runs the backward pass, and the printed values are the gradients for w and b.
- x has no
requires_gradflag, so the graph does not treat it as a parameter - w and b carry
requires_grad=True, which is what triggers recording - z = w * x + b builds the net input as a node that points back to w, x, and b
loss.backward()walks the recorded graph from the loss back toward the parameters
Approximately -0.187 for w.grad and -0.094 for b.grad (PyTorch prints more decimal places)Partial Derivatives Versus the Gradient
Partial derivatives are the building blocks of the backward pass. Each one measures how a single quantity changes with respect to one of its inputs while the other inputs are held in place. In a computation graph, each link between two nodes contributes one partial derivative, and the chain rule combines these pieces along a path. A gradient is what you obtain for the parameters when these pieces are assembled. The distinction is one of scope: a partial derivative describes one link, while the gradient describes the effect on the loss for each parameter.
| Aspect | Partial derivative | Gradient |
|---|---|---|
| Scope | One link between two nodes in the graph | The collection of effects on the loss for every parameter |
| What it measures | How one quantity changes with respect to one of its inputs, others held in place | How the loss changes with respect to each weight and bias |
| How it is produced | Read directly from a single edge of the graph | Assembled by chaining partial derivatives along each path from the loss to a parameter |
| Used for | A building block of the backward pass | Telling the training procedure how to update each parameter |
Partial derivatives and gradients in a computation graph
- 1The left side is the gradient of the loss with respect to the weight, one entry of the gradient.
- 2The first factor is the partial derivative of the loss with respect to the output a, which is the link from the loss back to the output.
- 3The second factor is the partial derivative of the output with respect to the net input z, which is the link through the sigmoid.
- 4The third factor is the partial derivative of the net input with respect to the weight, which is the last link to the parameter.
- 5Multiplying the three partial derivatives along this path gives the gradient for w. The bias follows the same path, except its last factor is the partial derivative of z with respect to b.
Chain Rule Moving Backward Through the Graph
Calculation moves forward, from inputs toward the loss. Gradient calculation moves in the opposite direction. It starts at the loss and works backward through the graph toward the parameters. The diagram below shows the order in which the backward pass visits each link and which local derivative is multiplied in at each step.
Backward pass with the numbers from the forward pass
Using x = 2, w = 0.5, b = 0.1, the output a of about 0.7503, and the squared error loss L = (a - y)2 with y = 1, find the gradients of the loss with respect to w and b.
- 1Start at the loss
The derivative of the squared error with respect to a is dL/da = 2(a - y) = 2(0.7503 - 1), which is about -0.4995.
- 2Pass through the sigmoid
The sigmoid's derivative is da/dz = a(1 - a) = 0.7503 times 0.2497, which is about 0.1873. Multiply it into the running product: -0.4995 times 0.1873 is about -0.0936.
- 3Reach the bias
Since z = wx + b, the partial derivative dz/db = 1. So the gradient for b is about -0.0936 times 1, which is about -0.094.
- 4Reach the weight
The partial derivative dz/dw = x = 2. So the gradient for w is about -0.0936 times 2, which is about -0.187.
The gradient for w is about -0.187 and the gradient for b is about -0.094. These match the values printed by the code example.
The Graph Versus the Code That Builds It
The Python lines that compute z and a are instructions that run once per forward pass. The graph is the record those instructions leave behind in memory: a structure in which each result points to the values it was computed from. The code tells you how the graph is built. The graph tells you what the backward pass can reach. The two are related but they are not the same thing.
| Aspect | Code that runs the forward pass | Recorded computation graph |
|---|---|---|
| What it is | Lines of Python that perform operations | A record of which values produced which results |
| When it exists | Written before the program runs | Created as the forward operations execute |
| What it answers | How a prediction is calculated | Which values each result depends on, for gradients |
| Used by | The interpreter, as instructions | The backward pass, to find gradients |
The code and the graph it leaves behind
Rebuilding the Graph Across Training Iterations
The graph is dynamic, which means it is produced as the forward computation runs rather than being fixed in advance. Across training iterations the cycle is the same: a forward pass records a graph, a backward pass uses it to find gradients, and the parameters are then adjusted before the next forward pass. The diagram below shows this progression across two iterations.
The point to notice is that the second forward pass builds its own graph. The source describes graph building as something PyTorch does during computation when the flag is set, so the graph belongs to the computation that produced it. Do not picture one permanent graph that is simply extended forever.
Misreadings of a Computation Graph
Common mistake Confusing the graph with the code that builds it
What it looks likeThinking that deleting the line z = w * x + b removes the z node from a graph that already exists.
Why it failsThe code runs the operations, and the graph is the record they leave behind. The graph describes dependencies, not the source text.
Do this insteadTreat the graph as a structure of values and the arrows between them, and the code as the procedure that creates it.
Common mistake Assuming every number in the program is recorded
What it looks likeExpecting the gradient of the loss with respect to a plain input x that has no requires_grad flag.
Why it failsRecording depends on terminal nodes that have
requires_gradset to True. Unflagged values do not trigger the trainable part of the graph.Do this insteadCheck which tensors carry the flag. Gradients are available for the values you have marked as needing them.
Common mistake Reading the arrows as two-way links
What it looks likeReasoning that the output a can be computed before the net input z exists.
Why it failsThe arrows are one-way dependencies. A value can be computed only after the values it depends on are ready.
Do this insteadRead the arrows as telling you which values must be ready before each step can run.
Common mistake Placing the label inside the prediction
What it looks likeDrawing the true label y as an input to the net input z.
Why it failsThe label is not part of the prediction. It enters the graph only at the loss, where the output is compared with it.
Do this insteadDraw the label feeding the loss node at the far end of the graph.
Common mistake Using the words partial derivative and gradient interchangeably
What it looks likeCalling the derivative across one link the gradient of the whole model.
Why it failsA partial derivative describes one link. The gradient collects the effect on the loss for every parameter after the links are chained.
Do this insteadSay partial derivative when you mean one edge, and gradient when you mean the result for each parameter.
Predicting the Net Input
The graph gives you the rule for the net input, so you only need to apply that rule to different numbers. Try the next step yourself before reading the answer.
What do you think happens?
With x = 3, w = 0.2, and b = -0.5, what is the net input z?
Reveal answer
z = 0.1
Multiply the input by the weight: 0.2 times 3 is 0.6. Then add the bias: 0.6 plus (-0.5) is 0.1. The sigmoid would be applied next, but the net input is the value at the z node.
Using the same weight and bias, trace the forward pass for an input of x = 3 and a label of y = 0. Name each intermediate result in order, and say at which node the label first enters the computation. Then state which parameters would receive gradients if the weight and bias carry requires_grad.
Hints
- Compute z first, then apply the sigmoid to get a, then compare a with the label at the loss.
- The label is used only at the loss node, not at z or a.
Summary of the Computation Graph
- 1PyTorch records the operations that produce a loss in a computation graph, and it does so by default when a terminal node has
requires_gradset to True. - 2Gradients are found by applying the chain rule from right to left, from the loss back to the weight and bias, which is reverse-mode automatic differentiation or backpropagation.
- 3A partial derivative describes one link in the graph, while the gradient collects the effects on the loss for each parameter.
- 4A logistic regression forward pass multiplies the input by a weight, adds a bias, and applies a sigmoid to give a score between 0 and 1, and the label enters only at the loss.
- 5The graph is the record left by the forward code, and it is rebuilt during each forward pass rather than being one permanent structure.
At a glance
- PyTorch records the operations that produce a loss in a computation graph, by default when a terminal node has
requires_gradset to True. - Gradients come from applying the chain rule from the loss backward toward the parameters, which is reverse-mode automatic differentiation or backpropagation.
- A partial derivative is the local piece on one link, and the gradient is the collection of effects on the loss for each parameter.
- A logistic regression forward pass computes z = wx + b, then a =
sigma(z)between 0 and 1, and the label enters only at the loss. - The graph is a record left by the forward computation and is rebuilt during each forward pass, so it should not be confused with the code that builds it.