Gradients and Weight Updates in Neural Networks
Weights are what the network searches over, the loss is how the search is scored, and gradients are the signals that steer each weight adjustment. If those gradients are distorted in size, the search itself is disrupted.
Why a Bigger Network Is Not Automatically Easier
A common intuition in machine learning is that a bigger network should be able to learn more. Adding layers gives a network more room to represent complicated relationships. In practice, however, stacking many layers together does not simply make learning easier. Training a deep network can become unreliable, and the network may fail to learn the patterns that would let it make accurate predictions.
What do you think happens?
Suppose a signal is multiplied by 0.5 at each of 10 successive layers. What is its final size compared with the starting size?
Reveal answer
The signal is multiplied by 0.5 ten times, so it becomes $0.5^{10} = \frac{1}{1024}$ of its starting size, roughly one thousandth.
Each layer shrinks the signal by half, and the shrinking compounds. The same compounding applies in reverse when the factor is greater than one.
Training as a Search for Weights
Training a neural network is a search. The network has a set of parameters, called weights, and the learning process looks for a particular set of weights that minimizes the loss function. The loss function scores how far the network's outputs are from what they should be. A lower loss means the network is doing a better job on the training data.
The adjustments to the weights during this search are guided by gradients. A gradient carries information about how the loss would change if a weight were nudged slightly. You do not need a deep mathematical understanding of gradients to follow this material. The working idea is enough: gradients are the signals that tell the learning process which way to move the weights.
Weights are what the network searches over, the loss is how the search is scored, and gradients are the signals that steer each weight adjustment. If those gradients are distorted in size, the search itself is disrupted.
Tracing a Signal Through Many Layers
Now trace what happens to a signal as it passes through a stack of layers. Imagine a signal that is multiplied by a number, passed through a layer, and then multiplied again by the same kind of number at the next layer. The question is what happens to the size of that signal after many such steps.
Shrinking and growing a signal across ten layers
A signal starts at size 1 and passes through 10 layers. Each layer multiplies it by the same factor. Compare a factor of 0.5 with a factor of 2.
- 1Shrinking case
With a factor of 0.5, each layer halves the signal. After 10 layers the size is $0.5^{10} = \frac{1}{1024}$, so the signal is about one thousandth of its original size.
- 2Growing case
Reverse the factor to 2. After 10 layers the size is $2^{10} = 1024$, so the signal is about 1024 times its original size.
- 3What the comparison shows
Neither factor is extreme on its own. Depth is what amplifies the tendency, because the same small per-layer change compounds across the stack.
A factor below one drives the signal toward zero, and a factor above one drives it toward a very large value. The example is simplified, since real networks do not multiply by one fixed number, but the compounding logic is the same.
This compounding effect is the core reason deep stacks behave differently from shallow ones. Small per-layer changes do not stay small once they are repeated many times. Vanishing and exploding gradients are two versions of this same compounding problem, appearing in the training signals rather than in a single forward pass.
Vanishing and Exploding Failure Modes
The source names vanishing or exploding gradients as the problem that can make training deep networks challenging. These two failure modes are named together because they share a cause and a consequence. Both are problems with the size of the gradients that flow through the network during training. Both lead to unstable training dynamics.
Comparing the Two Failure Modes
The two failures point in opposite directions but have the same practical result for the learner: the network cannot reliably adjust its weights. A gradient that shrinks toward zero starves the early layers of learning signal. A gradient that grows without bound floods them with updates that are too large to be useful.
| Feature | Vanishing gradients | Exploding gradients |
|---|---|---|
| Gradient size | Shrinks toward zero | Grows very large |
| Effect on early weights | Barely adjusted | Erratically adjusted |
| Effect on training | Learning stalls | Training destabilizes |
| Shared outcome | Network fails to find weights that minimize the loss | Network fails to find weights that minimize the loss |
Two versions of the same compounding problem in the training signal
Symptoms of Unstable Training
The source describes the effects of these gradient problems in three connected ways. Each one is a symptom you could look for when a deep network refuses to learn well.
- 1Unstable training dynamics: the training process does not progress in a steady, predictable manner.
- 2Difficulty adjusting weights: the network cannot effectively adjust its weights, so updates do little or do too much.
- 3Failure to find minimizing weights: as a result, the learning process struggles to find a set of weights that minimizes the loss function.
The consequence the source emphasizes is a failure of learning itself, not merely a slower version of learning. A network with unstable gradients can end up with weights that do not capture the structure the task requires, so it may be unable to make accurate predictions or decisions.
Why These Symptoms Motivate Layer Normalization
Putting the pieces together gives the motivation for layer normalization. Deep networks can suffer from unstable training, and the gradients that guide weight updates can shrink or grow as they pass through layers. Any remedy therefore has to address that instability so that the network can again adjust its weights and find parameters that minimize the loss.
Mistakes Learners Make About Deep Networks
Common mistake Assuming more layers always means more learning
Why it failsThe source states that stacking many layers does not simply make learning easier, and training a deep network can become unreliable.
Do this insteadTreat depth as added representational room that must also be trainable.
Common mistake Thinking vanishing and exploding gradients are unrelated problems
Why it failsThe two are named together because they share a cause, which is the compounding of gradient size across layers, and a shared consequence of unstable training.
Do this insteadThink of them as two versions of one compounding problem, one shrinking and one growing.
Common mistake Believing the gradient is the same as the weight
Why it failsWeights are the parameters being searched over, while gradients are the signals that guide how those weights are adjusted.
Do this insteadKeep weights (what changes) and gradients (what steers the change) as separate ideas.
Common mistake Assuming a poorly trained deep network just needs more time, when the instability can prevent it from learning the underlying patterns at all
Practice Tracing a Gradient
A deep network has ten layers. Early in training, the gradient at the last layer is large, but by the time it reaches the first layer it is nearly zero. Which failure mode is this, what does it do to the early weights, and what symptom would you expect to see in training?
Hints
- Ask whether the gradient is shrinking or growing as it moves backward.
- Consider which weights are described as hardly adjusted in the source.
- Think about whether the training process is progressing in a steady way.
Key Takeaways
- Training a neural network is a search for weights that minimize the loss, and gradients are the signals that guide each weight adjustment.
- Signals compound across layers, so small per-layer changes can grow into very large or very small values in a deep stack.
- Vanishing gradients leave early weights barely adjusted, while exploding gradients produce erratic updates; both lead to unstable training.
- Symptoms of unstable training include an unsteady training process, difficulty adjusting weights, and failure to find weights that minimize the loss.
- These failures motivate layer normalization, which is introduced as a response to the need for stability in the training signal.