Context Vectors and Input Representations
The attention mechanism is the process, and the context vector representations are its output. When you read about attention, keep these two roles separate.
Why a Single Element Is Not Enough
Large language models work with collections of input elements, and each element has to be represented in a form the model can compute with. A representation that describes only one element on its own leaves out how that element relates to the rest of the input. That gap is the problem this part of the chapter addresses. Attention mechanisms are the technique introduced to close it, and their purpose is to take input elements and transform them into something richer.
The central idea is simple to state. An attention mechanism produces enhanced context vector representations of the input elements, and those representations incorporate information about all of the inputs, not just the one being represented. The rest of this article unpacks that statement, traces where the information goes, and shows how to avoid the most common misreadings.
What do you think happens?
Suppose a set of input elements passes through an attention mechanism. What does one resulting context vector representation contain?
Reveal answer
Information about all of the inputs, not only the element it represents.
A context vector representation incorporates information about all inputs. The element it represents is only part of what it carries.
Two States of an Input Element
It helps to compare the two states an input element can be in. Before attention, each element is represented on its own. After attention, the same element is represented as a context vector that carries information about all inputs. Each input element is a separate item before the step, and each becomes a context vector representation afterward.
The diagram shows the change for one element, but the same transformation applies to the whole collection. Every input element that passes through the attention mechanism moves from a standalone representation to a context vector representation.
What a Context Vector Contains
A context vector representation is the enhanced form of an input element produced by an attention mechanism. It incorporates information about all inputs, not only the element it represents.
The attention mechanism is the process, and the context vector representations are its output. When you read about attention, keep these two roles separate.
The word 'context' in context vector points at what the representation holds: the surroundings of the element within the input. A standalone representation describes an element by itself. A context vector describes the same element together with information drawn from the whole collection. That is why the source frames attention as a way of making representations richer rather than merely different.
Where the Information Flows
Tracing the information shows what the attention mechanism actually does. The inputs go in as a collection. The mechanism reads across that collection. Then it produces one context vector representation for each element. The information in each output comes from the inputs as a whole, not from a single source element.
The diagram makes one point clear. Each output has its own identity, since it is the context vector for a particular element, but every output is built from the same collection of inputs. The source does not describe the internal computation that produces these outputs, so the picture stays at the level of what information goes where.
Tracing Three Input Elements
Following Three Elements Through Attention
Three input elements, labeled Element A, Element B, and Element C, pass through an attention mechanism. What does the representation of Element B look like afterward?
- 1Start with three standalone representations
Before attention, each of the three elements is represented on its own. Element B's standalone representation describes only Element B.
- 2Pass the whole collection through attention
The attention mechanism takes all three elements as its input. It does not process Element B in isolation from A and C.
- 3Read the output for Element B
The output for Element B is a context vector representation. Because it incorporates information about all inputs, it carries information drawn from Element A and Element C as well as from Element B.
- 4Check the other two elements
Element A and Element C each also receive their own context vector, built from the same three inputs. Each output differs by which element it represents, but all three are built from the full collection.
Element B is now a context vector representation that incorporates information about Element A, Element B, and Element C together. Its standalone representation has been replaced by this richer form.
Misreadings of What Attention Pays Attention To
The phrase 'pays attention to' invites readers to imagine a spotlight that lands on one part of the input. The source's description does not support that picture. Attention produces a context vector for each element, and each one incorporates information about all inputs. Several common misreadings come from taking the phrase too literally.
Common mistake Attention selects one element and ignores the rest
What it looks likeReading the worked example, you conclude that Element B's new representation is just a copy of Element B with A and C dropped.
Why it failsThe context vector for Element B incorporates information about all inputs, not only the element it represents.
Do this insteadThink of each output as built from the whole collection, with a particular element as its label.
Common mistake Attention finds the single most important element in the whole input
What it looks likeYou assume the mechanism outputs one winning element, and every other element is then described relative to it.
Why it failsThe source describes a context vector for each input element, not a single chosen element for the whole input.
Do this insteadExpect one context vector per input element, each produced from the collection.
Common mistake The attention mechanism and its output are the same thing
What it looks likeYou say the context vectors are attention, as if the two words named one object.
Why it failsThe source states that the attention mechanism is the process and the context vector representations are its output.
Do this insteadCall the procedure the attention mechanism and the results the context vector representations.
| Common misreading | What actually happens |
|---|---|
| Attention selects one element | Each element's context vector incorporates information about all inputs |
| Attention picks out the most important element overall | Each input element gets its own context vector |
| Attention is the same thing as its output | Attention is the process; context vector representations are the output |
| An element keeps only its standalone meaning | Each element changes from a standalone representation into a context vector |
Misreadings of attention compared with what the source describes
Checking Your Understanding
Three input elements, A, B, and C, pass through an attention mechanism. In two or three sentences, explain what has happened to Element C's representation, and say what information its new form contains.
Hints
- Compare Element C's standalone representation with its context vector representation.
- Ask whether the new form includes information only about Element C.
- Recall the source's statement that attention is the process and the context vectors are the output.
Key Takeaways
- 1An attention mechanism transforms input elements into enhanced context vector representations.
- 2A context vector representation incorporates information about all inputs, not only the element it represents.
- 3Each input element changes from a standalone representation into a context vector that reflects the whole collection.
- 4The attention mechanism is the process; the context vector representations are its output.
- 5Attention does not select one element or the single most important element; it produces a context vector for each element.
At a glance
- An attention mechanism transforms input elements into enhanced context vector representations.
- A context vector representation incorporates information about all inputs, not only the element it represents.
- Each input element changes from a standalone representation into a context vector that reflects the whole collection.
- The attention mechanism is the process; the context vector representations are its output.