Vanishing Gradient Problem in Deep Neural Networks

Gradients are computed for each layer with loss.backward(), and a per-layer mean of absolute gradient values makes layers easy to compare.

Programming

Why Early Layers Can Go Quiet

When a deep neural network trains poorly, one of the first questions worth asking is whether the learning signal reaches the early layers at all. Training works by sending a loss value backward through the network so that every weight can learn how to change. If that backward signal fades as it travels, the early layers hardly learn, no matter how long training runs. This article teaches you to see that fading directly, by measuring it one layer at a time.

What do you think happens?

If the signal shrinks as it moves backward through a deep network with no shortcut connections, which layer would you expect to show the smallest gradient?

The output layer, where the loss is computed
A middle layer
The first layer, farthest from the loss

Reveal answer

Answer:

The first layer, farthest from the loss.

The gradient starts at the last layer, where the loss is computed, and shrinks as it moves back toward the first layer. The first layer receives the weakest signal.

Tracing the Backward Signal

Picture the learning signal as starting at the loss and travelling backward through the layers, one after another. Each layer passes a gradient back toward the layer before it. The article's approach is measurement-first: instead of starting from the chain rule mathematics, you run a small helper on a deep model and read one gradient number per layer. The aim is to see where the signal gets weaker as it moves away from the loss.

How does the gradient travel backward from the loss through each layer, and where does it get weaker at each step?

Measuring Gradients One Layer at a Time

The helper used in this article is called print_gradients. It takes a model and a sample input, passes the input through the model to get an output, and compares that output with a target. The target here is simply zero, chosen for simplicity. The helper builds a mean squared error loss that measures how far the output is from the target, and then calls loss.backward(). That single call asks PyTorch to compute the gradient for each layer's parameters. You never write the gradient formulas yourself; PyTorch does that work, which is why training deep networks is practical.

A layer's weight matrix holds many values, and each one has its own gradient. A 3 by 3 weight matrix, for example, produces nine gradient values. To compare layers with one another, the helper takes the absolute value of every gradient in a layer and then averages those absolute values. The absolute value removes the sign, so positive and negative gradients both count toward the size of the signal. The result is one number per layer, which is easy to compare across the depth of the network.

python

import torch

# Illustrative sketch of the per-layer measurement idea.
# After loss.backward() has been called, each weight parameter has a .grad tensor.
def mean_abs_gradients(model):
    report = []
    for name, param in model.named_parameters():
        if param.grad is not None and name.endswith('weight'):
            mean_abs = param.grad.abs().mean().item()
            report.append((name, mean_abs))
    return report

import torch

# Illustrative sketch of the per-layer measurement idea.
# After loss.backward() has been called, each weight parameter has a .grad tensor.
def mean_abs_gradients(model):
    report = []
    for name, param in model.named_parameters():
        if param.grad is not None and name.endswith('weight'):
            mean_abs = param.grad.abs().mean().item()
            report.append((name, mean_abs))
    return report

Notes on the code

named_parameters() walks every learnable tensor in the model, in layer order.
param.grad holds the gradient that loss.backward() computed for that tensor.
abs().mean() removes the sign and averages every gradient value in the layer into one number.

Reading the Baseline Report

The baseline run uses a model built without shortcut connections. The helper is applied to it with the same sample input, and the report lists one mean gradient for each weight-bearing layer. Read the report from the last layer to the first. The pattern is consistent: the gradients get smaller as the signal travels backward. This is the phenomenon called the vanishing gradient problem.

When the mean absolute gradient is printed for each layer, how do the values change as you move from the output layer back to the first layer?

Worked example

Deciding Whether Gradients Vanish

You have printed the mean absolute gradient for each weight-bearing layer of a deep model without shortcut connections. How do you decide whether the gradients are shrinking toward the first layer?

1
Order the layers
Start at the last layer, where the loss is computed, and list the layers in the direction the gradient travels backward.
2
Compare neighbours
Check each layer's mean absolute gradient against the one closer to the loss. A consistent drop as you move backward is the pattern of vanishing gradients.
3
Check the endpoint
Look at the first layer. If its value is far below the output layer's value, the signal has faded across the depth.
4
Avoid one-number conclusions
Do not decide from a single layer. The pattern across all layers is what matters.

Result

If the values fall steadily from the output layer toward the first layer, with the first layer nearly vanishing, the gradients are shrinking toward the first layer.

How a Shortcut Reshapes the Pattern

A shortcut connection is a link that lets the signal bypass part of the network, so that information and gradients can travel across layers through a second route in addition to the usual one. The source describes shortcut connections as allowing gradients to flow more consistently across layers. In this course's model, the shortcut behaviour is switched on with a single argument, use_shortcut=True, when the model is created.

The diagram below is not a full architecture. It shows only the idea that a shortcut gives the backward signal a second route alongside the usual layer path. The source does not describe the exact internal arrangement, so treat it as an intuition aid for the concept of multiple gradient routes.

How does the per-layer gradient pattern look different when a shortcut connection is added compared with a plain stack of layers?

With shortcut connections, the last layer still carries the largest gradient. The difference is what happens as you move toward the first layer: the value stabilizes instead of dropping away to almost nothing. So when you compare the two reports, the endpoint is the key comparison. The first layer's gradient in the shortcut version is no longer vanishingly small, even though the overall ordering from output to input still has the output layer on top.

Keeping Comparisons Fair

The source seeds the random number generator with torch.manual_seed(123) before it creates the shortcut model. Seeding makes the random starting weights repeatable, so running the helper again gives the same report. This matters for comparisons: when two runs differ, you want to know the difference came from the architecture and not from the random starting point.

1
Fix the random starting point so both models begin from repeatable weights.
2
Use the same sample input and the same target for both runs.
3
Read both reports in the same direction, from the last layer back to the first.
4
Walk through the comparison in a fixed order before drawing a conclusion from any single number.

Mistakes When Reading Gradient Reports

Common mistake
Reading the report from the first layer forward
Why it fails
The gradient travels backward from the loss, so the comparison is clearest when you start at the last layer and follow it back.
Do this instead
Start at the last layer, where the loss is computed, and move toward the first layer.
Common mistake
Judging vanishing gradients from one layer's value
Why it fails
A single number cannot show a direction or a trend across depth.
Do this instead
Compare the whole sequence of layer values and check whether they fall steadily toward the first layer.
Common mistake
Treating the sign of a gradient as meaningful for size
Why it fails
The helper takes absolute values so that positive and negative gradients both count toward the size of the signal.
Do this instead
Compare the mean absolute values, which measure signal size.
Common mistake
Comparing two models with different random starts
Why it fails
Differences may come from the random starting weights rather than from the architecture.
Do this instead
Seed the random number generator before creating each model so the comparison isolates the architecture.

Why This Measurement Carries Forward

The source treats consistent gradient flow as a building block for very large models. It notes that shortcut connections are a core part of large models such as LLMs, and that they help make training more effective when the GPT model is trained in a later chapter. The measurement practised here is the same kind of check that helps you judge whether a deep model is receiving a usable learning signal in its early layers.

Practice Reading a Gradient Report

MEDIUM

You print the mean absolute gradient for each layer of two models, A (no shortcut connections) and B (with shortcut connections), using the same seed and the same input. For model A, the values fall steadily from the output layer to the first layer. For model B, the output layer is still the largest, but the first layer is no longer nearly zero. Which model shows vanishing gradients toward the first layer, and which comparison in the report supports your answer?

Hints

Start at the output layer and follow the values toward the first layer.
Ask whether the first layer's value has collapsed or stabilized.
Remember that the output layer is the largest in both models.

Your attempt (not saved)

Key Takeaways

The vanishing gradient problem appears as gradients shrinking as they move backward from the last layer to the first in a deep network without shortcut connections.
The helper measures one number per layer by averaging absolute gradient values after loss.backward(), which makes layers easy to compare.
A shortcut connection gives the backward signal a second route; the output layer still has the largest gradient, but the first layer no longer collapses to almost nothing.
Read reports in the same direction, from the last layer to the first, and fix the random seed so that differences come from the architecture.

Related concepts

More concepts starting with V