Query Rewriting in Conversational Systems

Imagine asking a system a question about your inbox, your company's sales records, or a fast-moving news feed. The system does not read everything it owns. It first pulls out a set of documents that look promising, and it puts them in an order. That first ordering comes from the retriever. It is often good enough to start with, but it is not always good enough to use directly.

There are two common reasons the first ordering falls short. Sometimes the list is far longer than anything you can hand to a language model. Sometimes the top of the list is not the most useful text for the question at hand. A step called reranking takes the retriever's ordering and revises it. This article follows the reasoning behind that step, then shows how a related technique, query rewriting, prepares the question before retrieval even begins.

Trace what happens to a question as it moves through the system. Start with a whole collection of documents. The query goes first to a retriever. The retriever is cheap to run but not very precise, so it can scan a large collection quickly and collect candidate documents. The output of this first stage is a candidate list. It is much smaller than the collection, and its ordering is only rough.

Now follow the candidate list into the second stage. A more precise mechanism examines only those candidates and reorders them. This mechanism is more expensive per document, which is exactly why it is applied to a short list instead of the whole collection. The state of the data changes at each stage: the collection becomes a candidate list, and the candidate list becomes a reordered, shorter list that is ready to be used.

The two-stage design separates two jobs that pull in opposite directions. Covering a large collection needs speed. Ordering a handful of documents with care needs precision. Rather than forcing one mechanism to do both, the design lets each stage do the job it suits. The retriever handles breadth and the reranker handles care.

Reranking is especially valuable when you need to reduce the number of retrieved documents. That reduction might be driven by the size of the model's context window, or by a wish to reduce the number of input tokens you pay to process. In both cases the reranker is a filter-and-order step. It turns a broad but rough list into a narrow and carefully ordered one.

Relevance to the query is not the only thing that can decide a document's position. Documents can also be reranked by time, with more recent material given higher weight. This matters when the answer changes over time. A document that was highly relevant last month may be less useful today than a slightly less matching document from this morning.

Recency matters most in time-sensitive uses. The source names three kinds of application: news aggregation, a chatbot that answers questions about your emails, and stock market analysis. In each case the newest information is often the most actionable, so a time-based weighting nudges it upward. This weighting is a deliberate adjustment applied on top of whatever the retriever has already judged. For a question whose answer does not change over time, relevance alone may be the better guide.

Reranking for a search results page and reranking for a language model are related, but they care about different things. On a search results page, rank is the whole point. The first result and the fifth result are different experiences, and rank position is crucial. When documents are placed inside a model's context, the exact position of each item matters less. What matters first is whether the document is included at all.

Order does not stop mattering once a document is inside the context. The order of documents still affects how well a model can process them. Models may understand documents placed at the beginning and at the end of the context better than those in the middle. Once a document is included, the effect of its exact placement is smaller than the effect of rank in a search list. Inclusion is the larger decision; arrangement is the refinement.

The map above shows one arrangement of three included passages. The beginning and end positions are the ones the model tends to use well, so the strongest passages are placed there and the weaker passage sits in the middle. The arrangement does not change which passages are included. It only refines how the model reads them.

Query rewriting, also called query reformulation, query normalization, and sometimes query expansion, means changing or restating a query so that it retrieves better results. The need for it becomes obvious in a conversation. Consider a user who first asks when John Doe last bought something from the business. The assistant answers that John last bought a Fruity Fedora hat two weeks before the date given. The user then asks, 'How about Emily Doe?'

Read on its own, that second question is incomplete. It does not say what is being asked about Emily. A system that sends those exact words to a retriever has very little to work with. The meaning lives in the earlier turn of the conversation. Query rewriting carries that meaning forward.

The exact wording of a rewrite is only an illustration. What the technique requires is that the standalone query captures the intent of the follow-up. Once the query is complete, retrieval has a real target, and the rest of the pipeline can work on documents that actually match what the user meant.