SGD and Backpropagation for Neural Networks

The Two-Direction Calculation

Backpropagation calculates the gradient of a loss with respect to a neural network's weights for one example (x, y). It does this by evaluating the network in two directions. The forward pass moves from the input toward the output and records the quantities produced by each layer. The backward pass starts at the output and moves toward earlier layers, combining the information needed to compute derivatives.

Forward Pass State

Think of the network as a sequence of composed functions. The input is the initial output, written as o_0 = x. At layer t, the network first forms a pre-activation vector by multiplying the layer's weight matrix by the previous output: a_t = W_(t-1)o_(t-1). It then applies the activation function to produce o_t = σ(a_t). The forward pass repeats this process from the input toward the final output.

Backward Error Signals

The backward pass begins at the final layer. For the squared loss used in the source material, the final error vector is δ_T = o_T - y. Earlier error vectors are then computed by combining the next layer's weights, the next layer's error vector, and the derivative of the next layer's activation. This recursion moves the loss information toward the first layer.

The chain rule explains why an early weight cannot be differentiated by looking only at its immediate layer. A change in that weight changes the next layer's output, which changes later outputs, which changes the loss. The derivative of the full network is therefore assembled from the derivatives of these successive transformations, in the appropriate order.

δ_T = o_T - y

Jacobian View

A Jacobian organizes partial derivatives by output component and input variable. Its shape records how many outputs depend on how many inputs. This makes it useful for describing the derivative of a layer whose input and output are vectors rather than single numbers.

An Edge Gradient

Consider the weights connecting layer t - 1 to layer t. The layer computes a_t = W_(t-1)o_(t-1), followed by o_t = σ(a_t). When differentiating with respect to the weights in this particular matrix, the earlier output o_(t-1) is treated as fixed. The derivative for an edge is then assembled from three local pieces: the error at the destination, the activation derivative at the destination, and the output at the source.

Gradient Discrepancy Diagnosis

A gradient mismatch should be investigated as a sequence of checkpoints. First inspect the forward pre-activations a_t and outputs o_t. Then inspect the final error δ_T and the recursively computed earlier error vectors δ_t. Finally inspect the derivative assembled for each edge. This order follows the dependency structure of backpropagation and helps distinguish a forward error from a backward error or an edge-calculation error.

• Checking only the final gradient value

  The gradient depends on forward pre-activations, forward outputs, error vectors, and the final edge assembly.

  Fix: Inspect the checkpoints in dependency order: a_t, o_t, δ_T, earlier δ_t values, and then the edge derivative.

• Ignoring stored forward quantities

  The edge derivative needs the source output and the destination pre-activation.

  Fix: Retain the forward values needed by the later backward calculation.

• Treating an early weight as affecting only its immediate layer

  A change in an early weight propagates through successive layer transformations before affecting the loss.

  Fix: Use the chain rule to combine the local derivatives across the composed functions.

SGD Connection

Backpropagation supplies the gradient of the loss with respect to the network's weights for one example. That gradient is the information an SGD procedure uses when working with the weights. The key boundary to remember is that backpropagation is the gradient-calculation procedure: it traces how the loss depends on the weights and produces the corresponding derivatives.

Practice Check

Key Takeaways

1. The forward pass computes each layer's pre-activation and output, while the backward pass computes error vectors and partial derivatives.
2. The chain rule connects local derivatives across the successive functions that compose a neural network.
3. A Jacobian organizes partial derivatives by output component and input variable; its shape records the relationship between the numbers of outputs and inputs.
4. An edge gradient depends on the destination error, the destination activation derivative, and the source output.
5. When a gradient is unexpected, inspect forward quantities, error vectors, and final edge-gradient assembly as separate checkpoints.