Self-Attention in Transformer Language Models

Attention is the mechanism that lets a language model decide how much each token should draw on other tokens in the sequence. That flexibility is powerful, but it creates a problem when the model is meant to generate text from left to right. If every token could freely look at every other token, the model could peek at words that have not been produced yet.

Some LLMs read and generate text from left to right. The model produces one token, then uses what it has produced so far to produce the next one. At any position, the tokens that come later in the sequence are not yet available to the model in the way a finished sentence would be. The model is building the sequence one step at a time, so the future is genuinely not there to be read.

The causal attention mask exists to enforce this ordering inside the attention computation. Its effect is that a token's attention can only be spread over the tokens at or before its own position. The rule is a constraint on direction: information may flow from the past into the present, but not from the future into the present.

A helpful way to see the causal mask is to draw a grid. Each row is a query token, the token whose attention is being computed. Each column is a token it could attend to. Following the example sentence 'The cat sat', the row for 'The' can see only 'The'. The row for 'cat' can see 'The' and 'cat'. The row for 'sat' can see all three tokens, because 'sat' is the last position and every earlier position lies at or before it.

Read the grid row by row and the pattern is clear. Each row includes everything up to and including its own position, and nothing after it. The visible sets are nested: each row sees one more token than the row above it, so the view of the past is complete at every step and the view of the future is always empty.

Attention weights describe how strongly one token attends to each other token. The causal attention mask sets the weights that point toward future positions to zero, so those positions contribute nothing to the result. Positions that are not in the future remain available and keep their weights. Because the masked weights are zero, a blocked future token has no influence on the output for that query.

The same rule explains what the model sees as generation proceeds. At the first step, only the first token exists in the visible context. Once that token is produced, the model can use it to produce the second token. At each new step the visible context grows by exactly one token, and every token already visible remains visible. Earlier steps are never revised by later ones, and later tokens are never visible to earlier steps.

The causal attention mask is not the only mask in the process. A dropout mask is used in addition to causal masking, and its purpose is to reduce overfitting by randomly dropping out attention weights. The source describes the two masks as applied together rather than as alternatives. The dropout mask is layered on top of the causal masking, not used in its place.

The order matters for understanding what each mask is responsible for. The causal mask settles which positions are blocked because they lie in the future. The dropout mask then acts on the weights that remain, randomly removing some of them so the model does not depend too heavily on any single attention pattern. The source frames this as addressing overfitting, which is a concern about the model's reliance on particular patterns rather than about the direction of generation.

A useful way to keep the two masks apart is to ask what question each one answers. The causal mask answers which tokens a position is allowed to see, given the direction of generation. The dropout mask answers how to keep the model from relying too heavily on any particular attention pattern. One concerns access and direction; the other concerns overfitting.