Stacking Single-Head Attention Layers

The current MultiHeadAttentionWrapper processes its heads sequentially through a list comprehension in the forward method.

Programming

Where the Wrapper Starts

Up to this point, a MultiHeadAttentionWrapper has been built by combining several single-head attention modules. Each of those modules is a complete attention unit that can process an input on its own. The wrapper gathers them together so that one call can produce the output of many heads. The exercise this article belongs to asks you to change what that combined output looks like, and its heading points at the target: returning two-dimensional embedding vectors.

Before changing anything, you need to be precise about how the current wrapper runs. This article traces that run step by step, then looks at the change the source proposes: computing all heads at once rather than one after another. The goal is to understand the forward pass well enough that a change to the output does not break the role each head plays.

How the Current Forward Pass Runs

The current MultiHeadAttentionWrapper processes its heads sequentially through a list comprehension in the forward method. The forward method passes the input through each head in turn, using a list comprehension of the form head(x) for head in self.heads. Every head receives the same input x, but the heads are evaluated one after another.

python

class MultiHeadAttentionWrapper:
    def forward(self, x):
        # each head is evaluated in turn, one after another
        outputs = [head(x) for head in self.heads]
        # the parallel version would compute all head outputs in one
        # matrix operation; the source does not give that code
        return outputs

class MultiHeadAttentionWrapper:
    def forward(self, x):
        # each head is evaluated in turn, one after another
        outputs = [head(x) for head in self.heads]
        # the parallel version would compute all head outputs in one
        # matrix operation; the source does not give that code
        return outputs

Notes on the code

The list comprehension visits self.heads in order, so head 1 finishes before head 2 starts.
The same x is passed to every head.

Tracing One Pass Through the Loop

What do you think happens?

When the wrapper runs with three heads, what does the second head receive as its input?

Only the output of the first head
The same input x that the first head received
An empty input, because the first head already consumed it

Reveal answer

Answer:

The same input x that the first head received

Every head receives the same input x. The heads are evaluated in sequence, but the input is not handed from one head to the next.

Worked example

Following the list comprehension for three heads

Trace what happens when forward is called with input x on a wrapper that holds three heads, named head 1, head 2, and head 3.

1
Start the comprehension
The loop reads self.heads in order and takes head 1 first.
2
Evaluate head 1
Head 1 is called with x, and its result is stored as the first element of the output.
3
Evaluate head 2
Only after head 1 finishes does the comprehension call head 2 with the same x.
4
Evaluate head 3
Head 3 is called last with the same x, completing the sequence.

Result

Three head outputs, each produced by a separate step, in the order the heads are listed.

Why Sequential Order Limits the Wrapper

The wrapper is correct in the sense that it produces the outputs of all heads. The feature that matters for the next change is the way it reaches that result: one head at a time, each in its own step. Because the design is built around that loop, each head is handled as a separate unit that the wrapper calls in turn.

This is the limit the source points at. If the wrapper must ask each head for its result in a separate step, then any change to how results are produced has to fit around that per-head calling pattern. The source proposes a different arrangement, which is the subject of the next section.

Computing All Heads Together

The improvement the source proposes is to process the heads in parallel. The concrete route it describes is to compute the outputs for all attention heads simultaneously by way of matrix multiplication. Instead of asking each head for its result in a separate step, the computation is arranged so that a single matrix operation produces the results for every head together.

Before

outputs = [head(x) for head in self.heads]

Each head is called in its own step, in list order.

After

# one matrix operation produces the results for every head together

The heads are treated as one batch of work, computed together.

What changes in the order of operations when heads move from a loop to a single batched computation?

The diagram above shows the shape of the idea in words: the input enters one shared computation, and the outputs for the individual heads come out of it together. In the sequential version, the input passes through one head, then the next, and so on. In the parallel version, the heads are treated as a single batch of work.

Weight Splits as an Organizing Principle

The next step in the source moves from the idea of parallel computation to a specific method: implementing multi-head attention with weight splits. The name carries the main point. Rather than keeping a separate, unrelated set of weights for every head as independent objects, the weights are organized so that they can be divided into portions, one portion for each head. Those portions can then be handled together in the matrix multiplication described above.

Which portion of the combined weights belongs to each head, drawn for three heads?

The diagram is a general picture of the weight-split idea, drawn for three heads as an example. Treat the portions as an organizing principle: a single set of weights is partitioned by head so that all heads can be computed in one pass. The source names this method but does not walk through its details in this section, so do not assume a specific layout beyond the division into one portion per head.

Mistakes When Changing the Wrapper Output

Common mistake
Changing what a head computes when the goal is to change the combined output
Why it fails
The source says to change how the combined result is computed and to keep the role of each head unchanged.
Do this instead
Edit the code that combines the head results, and leave each head's own computation alone.
Common mistake
Treating the parallel rewrite as a change to the mathematical quantity the layer produces
Why it fails
The move to computing all heads at once is about how the computation is organized, not which quantity is produced.
Do this instead
Confirm that each head's output is the same kind of result before and after the reorganization.
Common mistake
Assuming the wrapper's output format can be changed without checking the shape of what the heads return
Common mistake
Editing the list comprehension in ways that alter which head receives which input, when every head should receive the same x

Practice: Reworking the Combined Output

MEDIUM

Using the forward method sketch from this article, describe in your own words how you would change the wrapper so that it returns two-dimensional embedding vectors. Name the one part of the code you would edit, and name the part you would leave unchanged. Explain your reason for each choice in terms of the role each head plays.

Hints

Look for the line where the head outputs are gathered into a result.
The source says the role of each head should stay the same.

Your attempt (not saved)

Key Takeaways

The current MultiHeadAttentionWrapper runs its heads sequentially through a list comprehension, and every head receives the same input x.
The proposed improvement computes the outputs for all attention heads simultaneously by way of matrix multiplication, so one operation produces results for every head together.
Implementing multi-head attention with weight splits organizes the weights so they can be divided into one portion per head and handled together.
When modifying the wrapper's output, change how the combined result is computed and keep the role of each head unchanged.

Related concepts

More concepts starting with S