Implementing a Compact Causal Attention Class

Causal attention already decides which tokens a model may look at. Each position can draw information only from itself and from earlier positions, and a mask enforces this by setting the disallowed attention weights to zero. That rule is fixed: the same positions are hidden every time.

Dropout adds a second masking step on top of the causal one. Instead of following a fixed rule about which positions are hidden, dropout chooses positions at random and zeroes them out. According to the source material, the purpose is to reduce overfitting during training. In this article you will run the operation, watch what it does to a matrix, and learn how to check whether a result looks right.

The source starts with a deliberately simple experiment. It applies PyTorch's dropout to a 6 x 6 tensor filled with 1s, with a dropout rate of 50 percent. Filling the tensor with 1s is a simplification that makes the effect easy to read: any value in the output that is not zero must have come from a surviving 1. A zero in the output is therefore always a dropped position, and a non-zero value is always a survivor.

The same reading applies to any attention weight matrix. Once you know which values are zero and which are not, you can see exactly what dropout did, which is the habit the rest of this article builds on.

It helps to think of dropout as a two-part operation. First, it generates a mask: a pattern of positions that will be dropped, chosen at random. Second, it applies that pattern to the matrix, zeroing the selected positions. The positions that are not dropped keep their values, although they are rescaled, as explained in the next section.

Because the mask is random, the zeros do not fall in any fixed place that you could predict from the rate alone. The rate tells you how many positions are expected to be dropped on average. It does not tell you which ones.

The diagram is a hand-made illustration, not the output of a real run. Notice that both runs zero three of the six positions, which matches the rate, while the specific positions differ between the runs.

Zeroing half the entries would shrink the total influence of the attention weights if nothing else changed. To compensate for the reduction in active elements, the surviving values are scaled up. The source describes this as multiplying each surviving value by the factor 1 divided by the dropout rate.

Here $w$ is the original value of a surviving weight, $r$ is the dropout rate, and $w'$ is the value after scaling. With a dropout rate of 0.5, the scaling factor is $\frac{1}{0.5} = 2$, so each surviving value is multiplied by 2. A dropped position is not scaled, because it is already zero.

The same operation can now be applied to the attention weight matrix itself rather than to a matrix of ones. The code below assumes that the variable attn_weights already holds the causal attention weight matrix from earlier in the chapter. The seed is set again so that the random mask is reproducible for this run.

The code shows the operation in its simplest form. The seed controls the mask, the 0.5 controls the rate, and the layer handles both the zeroing and the rescaling. Each of these three inputs is a place where a result can differ from what you expected, and the last section of this article returns to them.

When you read the printed matrix, look at three things. First, the causal zeros from earlier are still present, because dropout does not undo them. A position that was already zero stays zero. Second, some rows lose most or all of their nonzero entries while others keep a few. Because the mask is random, this unevenness across rows is expected rather than a sign of a bug. Third, the surviving entries are no longer their original values but have been multiplied by 2 at a rate of 0.5.

The 3 x 3 grid above is a hand-picked illustration of the pattern, not a real run. Before dropout, the upper-right zeros come from the causal mask. Dropout then zeroes two more cells, the 0.4 and the 0.3, and doubles the survivors: the 0.6 in row 1 becomes 1.2 in row 2 only if it is the kept cell, so the diagram shows the same kind of change at each cell. Compare any nonzero cell in the after grid with its before value and you should find either a zero or exactly double the original.

Most confusion about dropout comes from a mismatch between what you expect the output to be and what the operation actually does. When a result looks wrong, follow the operation in order and check each step. The seed fixes the random mask. The mask chooses positions to drop. The rate sets how many positions are expected to be dropped. The scaling step then multiplies the survivors by 1 divided by the rate.

A wrong result can come from any step. If the seed changed, the mask changed, so the positions differ even though the rate is right. If the rate is wrong, the fraction zeroed is off. If the scaling is missing, the surviving values stay at their original size and look too small. If the causal mask was applied after dropout instead of before it, the layer treats its zeros differently from the intended order. Check each of these in turn rather than only the final numbers.