How Neural Networks Learn: Loss Functions and Optimization

A common intuition in machine learning is that a bigger network should be able to learn more. Adding layers gives a network more room to represent complicated relationships. In practice, however, stacking many layers together does not simply make learning easier. Training a deep network can become unreliable, and the network may fail to learn the patterns that would let it make accurate predictions.

The source material names two specific problems behind this difficulty: vanishing gradients and exploding gradients. This article builds up to those problems step by step and then shows why they create the need for layer normalization. The goal is to understand the problem first, so that layer normalization is understood as a response to something real rather than an arbitrary addition to a network.

Training a neural network is a search. The network has a set of parameters, called weights, and the learning process looks for a particular set of weights that minimizes the loss function. The loss function scores how far the network's outputs are from what they should be. A lower loss means the network is doing a better job on the training data.

The adjustments to the weights during this search are guided by gradients. A gradient carries information about how the loss would change if a weight were nudged slightly. In other words, gradients are the signals that tell the learning process which way to move the weights. You do not need a deep mathematical understanding of gradients to follow this material; the working idea is enough.

Picture a signal that is multiplied by a number, passed through a layer, and then multiplied again by the same kind of number at the next layer. The question is what happens to the size of that signal after many such steps. This compounding effect is the core reason deep stacks behave differently from shallow ones. Vanishing and exploding gradients are two versions of this same compounding problem, appearing in the training signals rather than in a single forward pass.

Read the diagram from the output toward the input. The gradient starts at full strength at the output and is weakened by a factor at every layer it passes through. By the time it reaches the earliest layers, it has been multiplied many times, so it is far smaller than where it began. The same arithmetic can work in the other direction, which is the origin of the exploding case described below.

The source identifies vanishing or exploding gradients as the problem that can make training deep networks challenging. In the vanishing case, the gradient reaching earlier layers becomes so small that those weights are hardly adjusted, so the network struggles to learn. The weights near the input receive almost no useful correction, even though the search for lower loss depends on them.

The example is deliberately simplified. Real networks do not multiply by one fixed number, but the compounding logic is the same. It explains why depth amplifies whatever tendency a gradient has to shrink.

In the exploding case, the gradient grows so large that the weight adjustments become erratic and destabilize training. Instead of nudging weights toward lower loss in a controlled way, the updates swing wildly. The search does not settle down, and the network cannot reliably find a set of weights that minimizes the loss function.

Reverse the factor in the previous example to 2 and the same arithmetic gives about 1024 at the first layer, which is the exploding pattern. The gradient is not small but enormous, and each weight update inherits that size. Both failure modes come from the same compounding arithmetic; only the direction differs.

The source describes the effects of these gradient problems in three connected ways. First, they lead to unstable training dynamics, meaning the training process does not progress in a steady, predictable manner. Second, they make it difficult for the network to effectively adjust its weights. Third, as a result, the learning process struggles to find a set of weights that minimizes the loss function.

The consequence the source emphasizes is a failure to learn the underlying patterns in the data. The network may then be unable to make accurate predictions or decisions. This is a failure of learning itself, not merely a slower version of learning. A network with unstable gradients can end up with weights that do not capture the structure the task requires.

Putting the pieces together gives the motivation for layer normalization. Deep networks can suffer from unstable training, and the gradients that guide weight updates can shrink or grow as they pass through layers. Any remedy has to address that instability so that the network can again adjust its weights and find parameters that minimize the loss.

The need, then, is for stability in the training signal. Layer normalization is introduced to answer that need. Understanding the vanishing and exploding problems first makes the technique a response to a concrete failure rather than an unexplained step in the network's design.