Input Representation in Language Models

Read this sentence: The trophy would not fit in the suitcase because it was too big. Ask yourself what the word it refers to. Most readers instantly link it to the trophy, not the suitcase, even though both nouns appear earlier. You did not reach that answer by reading each word in isolation. Your understanding depended on which earlier words mattered for the word it, and which ones could be set aside.

Large language models face a version of this same problem constantly. To produce a sensible continuation of a text, a model has to decide which parts of the input deserve the most weight at that moment. This article traces how the architecture behind modern LLMs makes that kind of selective focus possible.

When you hear that a model is a transformer, read that as a statement about how the model is built, not about what task it performs. The architecture describes the structure that processes the input; it does not by itself say whether the model is answering questions, translating, or continuing text.

The source gives the transformer a specific job in a chain of events. Input comes in, the transformer lets the model pay selective attention to different parts of that input, and the model then makes its predictions. The predictions depend on the transformer's selective focus, which is why the architecture is central to how LLMs work. The diagram below shows that order of events.

The word selective is doing real work in that definition. The focus is chosen, not spread evenly across everything. The source is also careful about what this does not mean: selective attention is uneven emphasis across the input, not ignoring most of it. Every part of the input can still take part in the prediction; the difference lies in how much weight each part receives.

The source ties this ability to a benefit. Because the model can focus on the parts of the input that matter, it becomes especially good at handling the nuances and complexities of human language. Human language often depends on relationships between distant words, on ambiguity, and on context. A model that weighs different parts of an input differently has a better chance of capturing those relationships than one that treats every part the same.

Follow a single prediction from start to finish. Assume a short piece of text has been given to the model as input. The input is not used in a single pass without distinction. The transformer lets the model direct its focus toward some portions of that input more than others. Only after that selective focus has shaped the model's internal processing does the model produce its prediction.

Notice what changes between the start and the end of this trace. The input stays the same, but the model's emphasis on different parts of it shifts depending on the prediction being made. A different prediction over the same text can lead the model to attend to a different set of parts. This is the core difference between selective attention and treating all input uniformly.

Return to the trophy sentence and apply the tracing idea to the word it. The prediction that matters is which earlier noun the word it points to. Two nouns are available, and the source's point is that they do not receive equal consideration. Readers direct more focus to the trophy, because the surrounding meaning makes the trophy the thing that could be too big, and they set the suitcase aside for this purpose.

Human language often depends on relationships between distant words, on ambiguity, and on context. The trophy sentence shows this: the link between it and its referent depends on a relationship that runs across several words. A model that can weigh different parts of its input differently has a better chance of capturing relationships like this than one that treats every part of the input the same, which is the benefit the source associates with selective focus.