Chunking Strategies for Documents

When a long document is split into smaller chunks, each piece becomes a separate unit that a retrieval system can return. That split is useful, but it carries a cost. A chunk is cut out of its surroundings, so it may no longer say what it is about. A paragraph that reads 'Restart the service and check the logs' is perfectly clear to someone who has just read the troubleshooting guide. Pulled out alone, it does not mention which service, which problem, or which product it belongs to.

Consider a support knowledge base. One chunk, taken from a longer article, reads: 'Open the security page, choose Reset, and follow the link sent to your inbox.' A customer types the query 'how do I reset my password?' For a person who reads the whole article, the chunk is clearly relevant. For a retriever, however, the match depends entirely on how well the query can be connected to the chunk's own content. The chunk never uses the word password.

The response to this problem is called contextual retrieval. Instead of accepting each chunk exactly as it was cut, information is added to it, so that the retriever has more to match against and the reader has more to understand. The rest of this article follows one chunk through that process and examines three kinds of added information: metadata, questions the chunk can answer, and context borrowed from the original document. It then looks at how AI models can generate a short context to tie the chunk back to its source.

It helps to think of augmentation as a sequence of steps that a chunk passes through between the original document and the form in which it is stored for retrieval. The source describes three kinds of added information and one generated addition. The order shown in the diagram is a teaching sequence that makes the progression easy to follow; the source does not present a fixed order. What matters is that each step adds something the bare chunk lacked before it becomes retrievable.

Notice that the chunk's own text stays in the middle of the process. Augmentation does not replace the passage. It wraps the passage in labels, likely questions, and references to the larger document, so that a query has more places to land and a reader has more to go on.

The simplest kind of augmentation attaches metadata to a chunk. Metadata is information about the content rather than the content itself. For retrieval, the most common forms are tags and keywords. Because these labels are stored alongside the chunk, a search can match on them directly, without needing the wording of the chunk body to line up with the wording of the query.

Extracted entities deserve a special note. Because they are extracted automatically, the labels are generated from the text itself. They do not depend on someone writing them by hand for every chunk, which matters when a collection holds many fragments.

Specific technical terms are a strong case for this technique. Suppose a document contains the error code EADDRNOTAVAIL (99). If that code is added to the document's metadata, a user who searches for the code can reach the document through that keyword. The source makes the important point that this works even after the document has been converted into embeddings, so the keyword remains a direct route to the content.

The source also gives examples for other content types. For ecommerce, a product can be augmented with its description and its reviews, so that a search for a feature mentioned only in reviews can still find the product. Images and videos can be queried by their titles or captions, because those short texts stand in for what the media contains.

A different kind of augmentation attaches to each chunk the questions it can answer. Users rarely search with the wording of a document. They ask questions, describe symptoms, or use informal phrasing. If the chunk is stored with the questions it answers, the search can match the user's phrasing directly to those questions rather than hoping the chunk's body uses the same words.

Metadata and questions add information about a chunk from outside its text. A third approach adds information from the document the chunk came from. The motivation is the problem described at the start: when a document is split into several chunks, some of them may lack the context needed to say what they are about. A chunk that begins in the middle of a procedure may never mention the procedure's name.

The remedy is to augment each chunk with context from the original document, such as that document's title and summary. The title names the topic of the whole document, and the summary describes what the document covers. Attaching these to each chunk means that a fragment carries a reference to the larger whole it came from, so the reader can place it within a familiar subject.

Titles and summaries help, but a chunk about a particular step may need a sentence explaining what that step is for and where it sits in the larger document. Writing such a sentence by hand for every chunk is slow, so the source describes using AI models to produce it automatically. The model is given both the chunk and the original document, so the generated sentence can connect the passage to its place in the whole.

The output is a short context. Anthropic, as the source reports, used AI models to generate a context usually between 50 and 100 tokens long. That context explains what the chunk is about and how the chunk relates to the original document. It covers detail that a title and summary cannot, because it is written for the specific passage rather than for the whole.