Term-Based Retrieval Models

Imagine asking a language model a question and giving it access to a large pile of documents to read first. A model can only take a limited amount of text into its context at once, so you cannot hand it everything. You need a rule for choosing which documents to include. This article covers two numbers that information retrieval systems use to make that choice, and a way of combining them into one score.

The starting point is a simple heuristic. When many documents contain the term you care about, include the ones that mention it most often. The reasoning is that a document which repeats a term many times is probably more relevant to that term than one that mentions it once. Before we can use that rule, we need a precise name for the repetition and a way to count it.

TF is a count that depends on only two things: the term and the one document you are looking at. If the word recipes appears three times in a particular document, then f(recipes, D) is 3 for that document. A different document might give a different count for the same word. Because the count is computed per document, it can be used directly as a ranking signal: a larger count means the document is more relevant to that term.

A term that appears almost everywhere is less informative than one that appears in only a few documents. Finding the word for in a document tells you very little, because almost every document contains it. Finding vietnamese in a document tells you much more, because few documents do. Inverse document frequency turns this intuition into a number.

In this formula, N is the total number of documents and C(t) is the number of documents that contain t. The source writes the value as a logarithm of the ratio N divided by C(t), but it does not state which base the logarithm uses, so this article does not assume one. The source's own worked example uses the plain ratio $\frac{N}{C(t)}$, which is the part to focus on first. Whatever the base, the logarithm increases as its input increases, so a term found in fewer documents still gets the higher IDF.

To get C(t), you need to know which documents contain each term. A simple way to picture this is a mapping from each term to the documents that list it. Once you have that mapping, the number of documents containing a term is just the number of entries attached to it. The index below is a small teaching example with N = 10 documents, where vietnamese appears in one document and recipes appears in five.

Reading the index this way makes the rarity idea concrete. The term for has the longest list of documents, so its C(t) is largest and its IDF is smallest. The term vietnamese has the shortest list, so it receives the largest IDF.

TF-IDF is an algorithm that combines term frequency and inverse document frequency. It produces a score for a document D with respect to a query Q. The query is treated as a list of terms t_1, t_2, and so on up to t_q. For each query term, you multiply that term's IDF by the term's frequency in the document, and then add up those products across all the query terms.

Read this formula from the inside out. For each query term t_i, take its IDF, which reflects how rare the term is across the collection, and multiply it by f(t_i, D), which reflects how often the term appears in document D. Each term therefore contributes a product that rewards both being rare overall and being repeated in this particular document. The summation symbol means you add these contributions together for every term in the query. The source calls this a naive definition, which signals that it is the simplest way to combine the two measures, not a final or tuned one.

Now suppose recipes appeared 10 times in D instead of once. Its contribution would rise to 2 x 10 = 20, which would now dominate the score. The score therefore balances how often a term appears in the document against how rare that term is across the collection. TF alone would have favored the repeated word, and IDF alone would have favored the rare word; the product and sum combine both signals.