Normalization in Neural Networks

Suppose you already know batch normalization, the traditional normalization method used in many neural networks. A natural question follows: if you want to normalize the activations inside a large language model, is there another option, and how does it differ? The answer depends on one decision that is easy to overlook. Normalizing always involves choosing a set of values to summarize together.

Think of a training step as a grid. Each row is one input from the batch, and each column is one feature of that input. Normalization needs statistics, and the direction in which you gather those statistics is the whole difference between the two methods. The grid below shows three inputs, each with three features, so you can see the two directions of gathering side by side.

Each cell holds one activation value. Each column is a feature, and each row is an input. Batch normalization and layer normalization differ only in which of these cells they treat as belonging to the same group when they compute their statistics.

The source describes the two methods as moving in opposite directions across this grid. Each method has its own pooling pattern, and that pattern determines what the statistics depend on.

The practical pressure behind this comparison is resources. LLMs often require significant computational resources, and the available hardware and the specific use case can dictate how large a batch you are able to use, both during training and during inference. When the batch is limited, the set of values that batch normalization pools together is limited too. A method whose behavior depends on how many inputs are processed together will therefore behave differently when the batch changes.

Layer normalization avoids this dependence. Because it normalizes each input independently of the batch size, the behavior for one input does not depend on which other inputs are sitting beside it. The source identifies this as offering more flexibility and stability in scenarios where resources are constrained, and it is particularly useful for distributed training or when models are deployed in environments with limited resources.

Given an input layout and a batch situation, the choice follows from which direction of gathering suits the constraint. The flowchart below captures the decision the source points toward: when the batch is limited, constrained, or spread across devices, the method that works within each input is the safer choice.

Normalization is only one part of the story in a transformer block. The source moves on to a small neural network submodule that is used as part of the transformer block in LLMs, called the feed forward submodule. Its first step is implementing the GELU activation function, which the source describes as playing a crucial role in this submodule.