Evaluating Embeddings for Semantic Retrieval

Imagine a retriever receives a query and hands back five documents. Some of them clearly address the question and some do not. If you wanted to judge that retriever, you would quickly run into a problem: a single number seems too blunt, because the retriever could be failing in two different ways. It might be filling its results with irrelevant material, or it might be missing relevant material that exists elsewhere in the database.

Context precision and context recall are the two metrics that separate these failure modes. Precision asks whether what the retriever returned is worth having. Recall asks whether the retriever found everything worth having. In this article you will compute both by hand, see exactly which counts each one uses, and learn why reading them together gives a fuller picture than either one alone.

Before defining either metric formally, follow one query from start to finish. Suppose a database holds 4 documents that are relevant to a query. The retriever returns 5 documents, and when you check them against the query, 3 of those 5 are relevant. The diagram below shows one possible arrangement of those five results. The positions of the relevant documents are illustrative; what matters is the count.

Notice what this tells you. Three relevant documents came back, so the retrieved set is partly good. But the database contained 4 relevant documents in total, which means one relevant document was missed. Keep these counts in mind, because each metric uses a different group of them. Precision looks at the five results. Recall looks at the four relevant documents in the whole database.

The denominator is the retrieved set, the list the retriever actually handed back. Precision never looks at documents the retriever ignored. Precision is also the more convenient of the two to compute: it only requires comparing the retrieved documents to the query, and that comparison can be carried out by an AI judge. You do not need to inspect the rest of the database to get a precision score.

Notice that the numerator is the same as in precision: the relevant documents that came back. Only the denominator changes. For recall, the denominator is the set of all relevant documents, which may be spread across the whole database, not just the retrieved list.

The two columns share the same numerator, 3 relevant retrieved documents, and differ only in what they divide by. Precision divides by the 5 documents the retriever returned. Recall divides by the 4 relevant documents that exist in total. Because the denominators differ, a retriever can score well on one metric and poorly on the other.

The denominator for recall is the set of all relevant documents, which may be spread across the whole database. Because of this, recall cannot be measured by looking only at what the retriever returned. To know how many relevant documents exist, you must annotate the relevance of every document in your database to the query. That is a much larger job than checking five results.

Precision never has to find the relevant documents that the retriever missed, because it only looks at what came back. Recall has to account for them. That is why the source notes that the precision comparison can be carried out by an AI judge without inspecting the rest of the database, while recall depends on annotation across the whole collection.

Both metrics depend on an evaluation set, which you build before you can score anything. The set contains a list of test queries and a set of documents. For each test query, every test document is annotated as relevant or not relevant. The annotation can be done by humans or by AI judges. Once those labels exist, you compute the precision and recall of the retriever on this evaluation set.

Precision and recall treat the retrieved documents as a group. They do not care whether the relevant document appeared first or last in the list. If the order of results matters to your system, precision and recall alone will not capture it. The source points to ranking-aware metrics for that case. For example, if more relevant documents should be ranked first, you can use NDCG (normalized discounted cumulative gain), MAP (Mean Average Precision), or MRR (Mean Reciprocal Rank).

The practical rule is this: use context precision and context recall when the question is which documents were found and how many were relevant. Reach for NDCG, MAP, or MRR when the question also depends on where in the list those relevant documents sit. Both groups of metrics are evaluations of the retriever in isolation, and neither is the final verdict on the system.