Efficient Computation of Masked Attention Weights

In causal attention, each token is allowed to gather information only from itself and from tokens that come before it. Tokens that come later in the sequence are supposed to stay invisible to it. This sounds simple, but it raises a question that often confuses learners: if future tokens are masked out and the remaining weights are renormalized, can those future tokens still shape the current token's result? The answer depends on what the softmax calculation actually does after masking, and this article builds that understanding one step at a time.

Leakage is a property of the final result. If a token at position 1 produces an output that depends on the value stored at position 2, which is a future position, then information has flowed backward in the sequence. The goal of causal masking is to make that flow impossible. The source also notes that the combined masking and renormalization effect prevents leakage not only from future tokens but from any other masked tokens, which is exactly the goal of causal attention.

Attention is computed for every pair of query and key positions, so the scores form a square grid. Each row belongs to a query token, the token whose output is being computed. Each column belongs to a key token, a position the query may look at. A causal mask marks which cells of this grid are allowed. In each row, the cells for the current position and earlier positions are visible. The cells for later positions are blocked.

Read the grid row by row. Query 0 sees only key 0. Query 1 sees keys 0 and 1 but not key 2. Query 2 sees all three keys because nothing in this sequence comes after it. The blocked cells are the future positions for each row, and those are the cells whose weights must end up contributing nothing to the output.

The process has a clear direction. Attention scores for every position are computed first. Future positions are then masked, which sets their weights to zero. Finally, the softmax is recalculated over what remains, so the surviving weights form the distribution the current token actually uses. The two steps have to be read together. Masking alone only removes future positions from the picture. Renormalization is what turns the remaining positions back into a proper set of attention weights that sum to 1.

The result is that the current token's output is built only from visible positions. After renormalization, the distribution is the same one you would get by running softmax only among the unmasked positions, so the masked tokens have no lasting effect on the final numbers.

Notice that the future token had the largest raw score in this row, and it contributed the largest intermediate weight. Its value would have dominated a naive mixture. Once it is masked and the rest are renormalized, its influence disappears entirely from the final distribution. This is the concrete meaning of 'judge leakage by the final distribution.'

The idea that trips learners up is the softmax denominator. Before masking, the denominator includes every position, including the future ones. This is where the mathematical structure of softmax shows up: the denominator is a single shared sum that normalizes all the weights. After masking, the future positions are removed and the softmax is recalculated, so the denominator now contains only the visible positions. The effect of the masked positions on the resulting weights is nullified. They do not contribute to the final score in any meaningful way.

The comparison shows why the denominator is not a leak. Before renormalization, the full denominator held the future term e^{^3}, and the masked weights therefore did not sum to 1. After renormalization, the denominator depends only on visible positions. The future term is gone from both the numerator and the denominator, so the final weights are the same as a softmax taken only over the allowed positions.