Evaluating Retrieval Algorithms

Every retrieval system has to answer a question that sounds simple: given a query, which stored items are the most relevant? The difficulty is that answering it quickly requires preparation. Before any query arrives, the system builds an index, which is a structure that organizes the stored items so that searching them later is fast. How much work goes into that structure is a choice, and that choice shapes everything that follows.

The central idea of this article is a trade-off. In retrieval systems, the more detailed the index is, the more accurate the retrieval process will be. The cost is that building the index becomes slower and more memory-consuming. Accuracy at query time is bought with effort at build time. Learning to see both sides of that exchange is what lets you choose an index that fits a real application, instead of defaulting to whichever one sounds most powerful.

Picture an index of potential customers. Each entry can hold only a name, or it can hold much more: the person's company, email address, phone number, and interests. Before reading on, think about what each choice does to the people who will search this index later.

Notice the two halves of the exchange. The rich entries pay off during searching, because the search has more attributes to work with. They pay during building, because every entry takes longer to create and occupies more storage. The customer example makes the trade-off visible before any algorithm name enters the picture.

The same reasoning applies to the indexes that retrieval systems build over vector data. Extra detail in the index gives the search more structure to exploit, and that structure is what helps it find relevant results. However, the extra structure has to be constructed first, and it occupies space once it exists. A more detailed index therefore moves the system along a dial: accuracy at query time rises, while build time and memory use rise with it.

Read the diagram column by column. The simple index is cheaper to build and smaller, and its accuracy is lower. The detailed index sits on the other side of the dial: it costs more to build and occupies more memory, and in return it retrieves more accurately. The diagram does not show a single winner. It shows a balance that you have to place for each application.

The source uses two named ANN indexes to show the trade-off in concrete form. HNSW is described as a detailed index. In general it provides high accuracy and fast query times, but it requires significant time and memory to build. LSH is described as a simpler index. It is quicker and less memory-intensive to create, but the queries it answers are slower and less accurate. The two indexes sit at opposite ends of the same dial.

Read each row across the two columns. HNSW pays at build time, with significant time and memory required, and in return answers queries quickly and accurately. LSH pays at query time, since its queries are slower and less accurate, and in return it is quicker and lighter to create. The source compares these behaviors at the level of cost and results. It does not describe how either index is constructed internally, so the comparison here stays with what each index costs and how it behaves when queried.

Trade-offs become easier to feel when you follow a single index through its life rather than comparing it at one moment. The source describes a team that maintains a catalog that changes every day. Its index goes through several phases, and each phase stresses a different metric.

How do you know whether an index sits at the right point on the dial? The ANN-Benchmarks website compares different ANN algorithms on multiple datasets, and it uses four main metrics that take the trade-off between indexing and querying into account. Each metric captures a different side of the picture. Together they let you judge an algorithm on what it costs to build, how fast it answers, how well it finds the true neighbors, and how much space it takes up.

The first metric, the fraction of nearest neighbors found, is the one most directly tied to the detailed-versus-simple distinction, because it measures accuracy. The other three are costs or capacities: query throughput, build time, and index size. A complete comparison has to look at both sides. An algorithm that scores well on accuracy alone may still fail the build-time or storage limits of a real system.

The same discipline of measuring what matters extends beyond the index to the whole retrieval-augmented generation pipeline. The source's summary is that the quality of a RAG system should be evaluated both component by component and end to end. Component-wise evaluation looks inside the pipeline, checking the parts that feed the final answer. End-to-end evaluation looks at the system's final outputs as a whole.

Both levels are needed because each can hide a problem from the other. Evaluating only the final answers can hide a weak retriever, since a good-sounding answer may come from luck or from a generator that compensates. Evaluating only the retriever can miss a generator that fails to use good retrieved material. Checking the components and the whole together gives you a picture that neither view provides alone.