Calculus

From Loss to Learning Information

A neural network can be viewed as a sequence of connected operations. One operation produces an output that becomes the input to another operation, and the final result produces a loss value. Backpropagation is the algorithm that starts with this final loss and works backward through the connected operations to compute gradient values.

The Forward Route and Its Reverse

Imagine following the network in the order in which its operations are connected. An earlier operation produces a value, a later operation uses that value, and the final operation produces the loss. This is the forward nesting order: later operations depend on results produced earlier in the network.

Why the Chain Rule Is Necessary

A network is not one isolated operation. It is built from connected operations, so an earlier value can affect the loss through the later operations that use it. To determine the effect of that earlier value, backpropagation must combine the derivative information from the later part of the network with the local derivative of the operation currently being examined. This is the role of the chain rule.

For a nested network, the backward traversal follows the reverse of the forward nesting order. At each operation, the information already obtained from later operations is combined with the local derivative of the current operation. Repeating this process carries gradient information toward earlier operations and their parameters.

Tracing One Parameter's Influence

The important idea is not to inspect a parameter in isolation. Its contribution is determined by the entire connected route from that parameter to the final loss. Every later operation on that route contributes local derivative information that must be incorporated during the backward traversal.

Checking a Backpropagation Trace

1. Begin at the final loss.
2. Inspect the operation immediately before the loss.
3. Move toward earlier operations in the reverse of the forward nesting order.
4. At each operation, check that later gradient information is combined with the operation's local derivative.
5. When a result diverges from what is expected, locate the first operation where that backward combination no longer follows the chain-rule structure.

Common Direction and Chain-Rule Errors

• Starting the backward calculation at the earliest operation.

  Backpropagation starts with the final loss and moves backward toward earlier operations.

  Fix: Begin at the final loss, then follow the reverse of the forward nesting order.

• Treating connected operations as if they were independent.

  The chain rule combines local derivatives of connected operations.

  Fix: At each operation, combine the derivative information from the later part of the network with the local derivative of the current operation.

• Stopping after examining the operation nearest to the loss.

  The purpose of backpropagation is to determine how the network's parameters contributed to the loss.

  Fix: Continue the reverse traversal until the relevant earlier operations and parameters have been reached.

• Looking only at the final result when checking a calculation.

  A divergence can be localized to the first operation where the backward chain-rule structure stops being followed.

  Fix: Check the traversal one operation at a time, beginning at the final loss.

Practice the Reverse Trace

What to Remember

1. Backpropagation is an algorithm for computing gradient values in a neural network.
2. A neural network can be understood as connected operations whose final result produces a loss.
3. Backpropagation starts at the final loss and moves backward from later operations toward earlier ones.
4. The chain rule combines local derivatives across connected operations.
5. A parameter's contribution to the loss is determined by tracing the backward route from the loss through the operations connected to that parameter.