Implementing Multi-Head Attention with Weight Splits

Up to this point, a MultiHeadAttentionWrapper has been built by combining several single-head attention modules. Each of those modules is a complete attention unit that can process an input on its own. The wrapper gathers them together so that one call can produce the output of many heads. The exercise this article belongs to asks you to change what that combined output looks like, and the heading of the exercise points at the target: returning two-dimensional embedding vectors.

Before changing anything, it helps to be precise about how the current wrapper runs. The question this article answers is how the work gets organized inside the forward method, because that organization determines how easy it is to change the output later. The rest of the article traces the current flow, then the proposed parallel flow, then the weight-split method that makes the parallel flow possible.

The current forward method passes the input through each head in turn, using a list comprehension of the form head(x) for head in self.heads. Every head receives the same input x, but the heads are evaluated one after another. Read the code below and notice that the loop is written inside the comprehension itself. Each pass through the comprehension asks one head for its result before moving on to the next head.

The wrapper is correct in the sense that it produces the right outputs. The limitation is in how it reaches the result. Each head is a separate step, so the structure of the forward method is built around a loop over heads. That structure is what the rest of this article examines.

The improvement the source proposes is to process the heads in parallel. The concrete route it describes is to compute the outputs for all attention heads simultaneously by way of matrix multiplication. Instead of asking each head for its result in a separate step, the computation is arranged so that a single matrix operation produces the results for every head together.

This change affects the shape of the forward pass, not the purpose of the wrapper. The wrapper still combines multiple single-head attention modules, and each head still plays the same role. What changes is that the forward method no longer needs a per-head loop to get its results. The head outputs come out of one shared computation.

The next step in the source moves from the idea of parallel computation to a specific method: implementing multi-head attention with weight splits. The name carries the main point. Rather than keeping a separate, unrelated set of weights for every head as independent objects, the weights are organized so that they can be divided into portions, one portion for each head. Those portions can then be handled together in the matrix multiplication described above.

The source presents the slices as an organizing principle. It names the method and describes its goal, but it does not walk through the details in this section. The key idea to hold onto is that a single set of weights is partitioned by head so that all heads can be computed in one pass. The partition is what lets one matrix operation serve every head.

Whether heads are computed one at a time or together, the wrapper must still deliver one combined result. The source's rule for modifying the output is that you change how the combined result is computed and keep the role of each head unchanged. In practice, this means each head's output is joined with the others to form the final output, while each head continues to do the same job it did before.

The joining step is the part you modify when the goal is a different output format. The small illustration below uses plain Python lists to show the idea. Each head contributes a short list for one token, and the combined output is their concatenation in head order.