Automatic Differentiation with Autograd
PyTorch is organized into three broad components: a tensor library, an automatic differentiation engine, and a deep learning library.
A Training Loop to Follow
PyTorch is a large library with many functions, classes, and modules, and a newcomer can easily feel lost on the first day. A useful way to get oriented is to stop trying to memorize every part and ask a simpler question instead: what broad jobs does the library perform? The source material answers this by grouping PyTorch into three broad components. Each one solves a different problem, and the three are designed to work together.
To make those jobs concrete, imagine you are training a small neural network to recognize handwritten digits. Before the network can learn anything, your images must be held in memory as numbers. The network multiplies those numbers by a set of adjustable weights, compares its output with the correct answer, and then nudges the weights so the next guess is a little better. Repeat this thousands of times and the network gradually improves.
What do you think happens?
In the digit-recognition loop, which PyTorch component works out how much each weight should change?
Reveal answer
The automatic differentiation engine (autograd)
The source assigns working out how much each weight should change to autograd. The tensor library holds the numbers and performs the multiplications, and the deep learning library organizes the layers, the error measure, and the update rule into reusable parts.
The Three Components at a Glance
PyTorch is organized into three broad components: a tensor library, an automatic differentiation engine, and a deep learning library. Each component has one main job. The list below names each component and states what it is responsible for.
- Tensor library: stores data as multidimensional arrays of numbers and performs computation on them, with GPU acceleration.
- Automatic differentiation engine (autograd): computes gradients for tensor operations so you do not have to derive them by hand.
- Deep learning library: provides modular, flexible, and efficient building blocks such as pretrained models, loss functions, and optimizers.
The three components are not three unrelated toolboxes. The deep learning library sits on top, and the tensor library and autograd form the foundation it relies on. Autograd does not compute gradients in isolation. It operates on the tensors and tensor operations that the tensor library provides, which is why the tensor library comes first when learning the components in order.
Tracing One Training Step
Return to the digit-recognition loop with the three roles in mind. A single training step touches all three components in sequence. Holding the images as numbers and multiplying them by the weights is the job of the tensor library. Measuring the error is the job of a loss function from the deep learning library. Working out how much each weight should change is the job of autograd. Applying those changes through an optimizer is again the deep learning library's job, now consuming the gradients autograd supplied.
Read the flow from left to right and notice what stays out of your hands. You supply the data and the model structure. The engine supplies the gradients, and the optimization step consumes them. The handoff from the operations to the gradients to the update is exactly where the source places autograd: it simplifies backpropagation and model optimization without a manual gradient calculation in the middle.
What Autograd Computes
Training a neural network means adjusting its parameters so that its outputs get better over time. To know which way to adjust each parameter, the training process needs gradients. A gradient describes how much a result changes when one of its inputs changes slightly. Gradients tell the model which direction to move its weights, so computing them is a central step in every training cycle, repeated for every batch of data the network sees.
Autograd, short for automatic differentiation, is PyTorch's system for producing those gradients. The word automatic is the key. You describe the computation you want, and the framework produces the gradients for that computation. Autograd computes gradients for tensor operations, so the person writing the training code does not need to derive a gradient formula for each operation and then write code to evaluate it.
A Three-Stage Calculation
Consider a very small calculation: multiply an input by a parameter, add a constant, then compare the result with a target value to measure how wrong it is. How does training learn from the final error?
- 1Stage one
The input is multiplied by the parameter. This is a tensor operation.
- 2Stage two
A constant is added to the result. This is another tensor operation that transforms the value produced by the stage before it.
- 3Stage three
The result is compared with the target value, producing a measure of error.
- 4Gradient question
Training needs to know how the final error responds to a small change in the parameter. That answer depends on every stage in between, which is the bookkeeping autograd performs.
Autograd supplies how the final error changes with respect to the parameter by following the chain of stages, so the training code never has to write that chain out by hand.
Backpropagation Versus Autograd
Backpropagation is the algorithm used to train neural networks. It describes how error information flows backward through a network so that every parameter can be updated. Autograd is PyTorch's automatic differentiation system, and it is what makes backpropagation convenient to use. The two terms are easy to blur, so it helps to keep them separate: one is the method, and the other is the machinery that supports the method in PyTorch.
| Aspect | Backpropagation | Autograd |
|---|---|---|
| What it is | An algorithm for training neural networks | PyTorch's automatic differentiation system |
| Main purpose | Pass error information backward so every parameter can be updated | Compute gradients for tensor operations automatically |
| Who writes it by hand | Not needed when autograd is used; the training code relies on autograd | Handled by PyTorch rather than the programmer |
| Relationship | The stage of training that autograd supports | The system that makes backpropagation convenient |
Backpropagation is the method; autograd is the PyTorch system that supports it.
Why Manual Gradients Would Be Hard
Without a system like autograd, a person building a network would need to derive the gradient formula for each operation by hand and then write code that evaluates those formulas. Computing gradients by hand becomes impractical once a model contains many operations. Automatic differentiation removes that manual step, which is why it matters for training at all, not only for convenience.
Common mistake Thinking autograd and backpropagation are two different training methods
Why it failsBackpropagation is the algorithm; autograd is the PyTorch system that carries out the gradient bookkeeping that backpropagation needs.
Do this insteadSay backpropagation when you mean the training method, and autograd when you mean PyTorch's tool that computes the gradients for it.
Common mistake Assuming the deep learning library computes gradients itself
Why it failsThe deep learning library supplies layers, loss functions, and optimizers. It depends on autograd for gradients and on the tensor library for the data.
Do this insteadLook to autograd for gradients and to the optimizer for applying the weight updates.
Common mistake Believing autograd can work without the tensor library
Why it failsAutograd operates on the tensors and tensor operations that the tensor library provides, so removing the tensor layer removes what autograd computes over.
Do this insteadKeep the three components in their layered relationship: the tensor library is the foundation for both autograd and the deep learning library.
Checking Your Understanding of the Training Step
A training step runs a forward pass that produces a prediction, computes a loss that compares the prediction with the target, computes gradients, and then updates the parameters. For each stage, name the PyTorch component that handles it: the tensor library, autograd, or the deep learning library.
Hints
- The tensor library stores data as arrays of numbers and performs the operations on them.
- Autograd is the component that computes gradients for tensor operations.
- Loss functions and optimizers are reusable building blocks in the deep learning library.
Key Takeaways
- PyTorch is organized into three broad components: a tensor library for arrays with GPU acceleration, an automatic differentiation engine (autograd) for gradients, and a deep learning library for modular building blocks.
- The components depend on one another in layers: the deep learning library uses autograd and the tensor library, and autograd operates on tensor operations.
- A single training step passes through all three components: tensors hold and transform the data, a loss measures error, autograd produces gradients, and an optimizer applies the weight updates.
- Backpropagation is the algorithm for training neural networks; autograd is PyTorch's system that computes the gradients that backpropagation requires, so you do not derive them by hand.