Dropout in Deep Learning

Imagine a training process where a model keeps finding one small group of internal units that carries most of the useful signal. Those units become a crutch. The model performs well on the examples it has seen, but it has learned to depend on a narrow set of pathways rather than on a broad, robust pattern. This is the situation that the technique called dropout is designed to discourage.

Dropout is a training technique in which randomly selected hidden layer units are ignored. By ignoring units at random, it helps prevent overfitting, because the model cannot rely too heavily on any specific set of units. This article first looks at dropout as it appears in ordinary deep learning, then moves into the attention mechanism of transformer architectures such as GPT. The goal is to understand not only what dropout does, but exactly where it sits inside attention and why one placement is chosen over another.

The key word in the definition is selection. The units are not chosen by a human, and they are not always the same units. Each training step draws a fresh random selection, so over many steps every unit is sometimes present and sometimes absent. No single unit can be guaranteed to be available, and therefore no single unit can become indispensable to the model.

This is why random ignoring is useful while the model is learning. Because the network never knows which units will be present on the next step, the weights are shaped so that they do not depend on fragile combinations of units. The network is pushed toward a broad, robust pattern instead of a narrow set of pathways. Dropout is therefore a regularization device that works through the learning process itself.

Dropout is a training-time device. Once training is finished and the model is used to produce outputs, dropout is disabled, so it does not affect the model once it is in use. The network then works with all of its units available. The randomness that helped shape the weights during learning has no role when the model answers new inputs.

The diagram below contrasts the two states. The same layer is involved in both, but the mask that decides which units are ignored behaves very differently. During training it is sampled fresh on each step. During use it is switched off.

Dropout is not limited to fully connected hidden layers. In the transformer architecture, including models like GPT, the attention mechanism can also use dropout. The source identifies two specific moments when this happens: after the attention weights have been calculated, or after those attention weights have been applied to the value vectors. In both cases, dropout is a random masking step inserted into the flow of attention.

The two placements answer one simple question: which quantity should be randomly ignored? Either the weights that decide how much each position attends to others, or the result obtained after those weights have been combined with the value vectors. The flowchart below follows the attention computation from raw scores to output and marks both candidate points. The first candidate is the one this course follows.

Read the flowchart as a sequence of choices rather than a sequence that always uses both options. In a given design, dropout can sit at option A, at option B, or in the relevant spot for that design. The source describes both as valid times to apply dropout in attention, so seeing them side by side makes the choice easier to understand.

Consider what happens when dropout sits at option A, directly after the attention weights are calculated. Each attention weight says how much one position attends to another. A dropout mask then decides, entry by entry, which weights are ignored. The example below uses made-up numbers for four attention weights and four value vectors. The numbers are illustrative only and are not taken from the source.

The diagram shows the state change for this one step. Weights 2 and 4 are the ones the mask ignores, so they are the entries that change. Weights 1 and 3 pass through untouched. Because the weighted sum over the value vectors is built from these weights, the output of attention shifts as well. A new mask on the next step would choose a different subset, which is why no single position can be relied on across training.

This course applies the dropout mask after computing the attention weights. The source describes that placement as the more common variant in practice, which is the reason this course follows it. The walkthrough above is therefore the placement the course uses: the mask acts on the weights that decide how much each position attends to others, before those weights are applied to the value vectors.

The other placement, option B, masks the result after the weights have already been combined with the value vectors. Both are valid according to the source. The practical reason the course prefers option A is that the source describes it as the more common choice, and the course adopts the same convention so that learners see the variant most often encountered in practice.