Positional Embeddings in Transformer Models

Suppose a language model can only take in a certain number of tokens at once, and you want it to read much longer documents or code files. The obvious plan is to gather longer documents, feed them to the model during finetuning, and let training do the rest. This article explains why that plan is incomplete. Extending how much a model can read at once is a different kind of change from most finetuning, and it reaches into the model's architecture, specifically the positional embeddings that record where each token sits.

The first guess goes like this: feed the model longer documents during finetuning, let it practice on them, and it learns to read further. Under this picture, only the amount of text in each training example changes, and the model's internal machinery stays the same. The belief is understandable, because most finetuning works that way. You change the data, run training, and the model's behavior shifts. Long-context finetuning is different, and the difference matters for anyone who wants to extend a model's reach.

The defining feature of long-context finetuning is that it typically requires modifying the model's architecture rather than only changing the training data. The source names a specific part that is usually adjusted: the positional embeddings.

Every token in a sequence occupies a position, and the model needs a way to represent where each token sits. Positional embeddings are the component of the architecture that handles these token positions. You can think of them as the model's record of where each token sits in the sequence. Each position in a short input has a slot in that record. When the input grows, more slots are needed.

The diagram is a simplified picture of the record. The first slots cover the positions the model was built for, and the added slots represent the wider range that extended context requires. The source does not describe a specific method for how the added positions are represented, so the diagram shows only that the record must grow to cover them.

The three pieces are tied together. The context window size sets how many token positions a model must handle. The positional embeddings represent those positions. Token positions are what each token in an input occupies. A longer sequence means more possible token positions, so the positional embeddings must be able to cover those additional positions. If the component was built only for shorter positions, feeding it longer inputs is not enough. The component itself has to be adjusted so it can represent the wider range.

The source gives a concrete case. Code Llama models were created from the base model Llama 2 using different finetuning techniques. Using long-context finetuning, the developers increased the model's maximum context length from 4,096 tokens to 16,384 tokens so it could accommodate longer code files. Code files are a natural motivation, since a single source file or a set of related files can run far longer than a short question or a paragraph.

The source places long-context finetuning alongside two other techniques, and the comparison clarifies what makes it distinct. Instruction finetuning is supervised finetuning on instruction data. Its responses can be open-ended, such as a book summary, or close-ended, such as a classification answer. The source notes that high-quality instruction data can be challenging and expensive to create, especially when instructions demand factual consistency, domain expertise, or political correctness.

Preference finetuning uses reinforcement learning to push the model toward responses that people prefer. It depends on comparative data, typically arranged as an instruction, a winning response, and a losing response. Long-context finetuning, by contrast, is aimed at a property of the model: how many tokens it can handle. It is not defined by the kind of data or the reward signal used during training. Neither instruction finetuning nor preference finetuning is defined by extending context length.

Extending context length is harder than many other finetuning techniques, and the resulting model might degrade on shorter sequences. Before extension, the model handles short inputs with the positional record it was built for. After extension, the same model is asked to represent a wider range of positions, and its behavior on short inputs is no longer guaranteed to stay the same. The source states the risk as a possibility, so the practical lesson is to check short-input behavior after a context extension rather than assume it is preserved.