Vector Representation of Documents

Suppose you have a collection of text documents and you want a computer to turn each document into a list of numbers. A simple approach is to count how often each word appears. That works, but it has a weakness. Common words such as the or and appear almost everywhere, so their counts tell you very little about what a particular document is about. A document that mentions the many times is not necessarily a document about the.

tf-idf is a method for scoring each word in a document so that words which are frequent in that document but rare across the collection receive higher scores. Before we look at how it works, try a quick prediction.

A vectorized document is a list of numbers with a fixed length. Every word in the vocabulary, which is the set of words the system knows about, is assigned one fixed position in that list. The document does not choose its own positions. Each position always refers to the same word, so two documents can be compared position by position.

The numbers that fill those positions come from the document itself. For each vocabulary word, the document contributes a value for that word's position. Words the document never uses still have a position, and that position simply holds a value for zero occurrences in the raw count version.

In this generated example, the vocabulary has four words, so every document becomes a vector with four positions. The document shown contains cat twice, sat once, mat once, and never mentions dog. Its vector is therefore determined entirely by those four positions, in that fixed order. A tf-idf representation follows the same layout, but each position holds a weight instead of a plain count.

A tf-idf score for a word $w$ in a document $d$ is computed from three quantities. Each one answers a different question about the word and the collection it lives in. Keeping them separate in your head makes the rest of the calculation much easier to follow.

The central idea is the direction of the effect. Hold the term frequency fixed, and look at how the score changes as the number of training documents containing the word grows. The score for a word that appears in few documents is larger than the score for a word that appears in many of them. The exact formula combines N, $N_{w}$, and tf together, and the precise numbers depend on that formula. What you should carry forward is the direction: a larger $N_{w}$ relative to N pulls the score down, and a smaller $N_{w}$ relative to N keeps it higher.

After the tf-idf representation is computed, the classes in this source also apply L2 normalization. L2 normalization rescales the representation of each document so that it has Euclidean norm 1. In other words, every document's vector is scaled to the same overall length, regardless of how large its raw values were.

The Euclidean norm of a vector is the square root of the sum of its squared values, written $\lVert v \rVert_2$. Dividing every value in the vector by this norm produces a new vector whose norm is 1. Only the overall length changes; the proportions among the values stay the same.

This is why document length stops mattering. A long document naturally contains more words than a short one, which tends to inflate raw counts. The source states that rescaling in this way means the length of a document, measured as the number of words, does not change the vectorized representation. Two documents that use the same words in the same proportions end up with the same normalized vector, even if one of them is much longer.

Grid search tries different combinations of settings and compares how well each one performs. When tf-idf is one of the steps being tuned, its calculation depends on statistics of the training data, such as N and $N_{w}$. Those statistics are only meaningful for the training data they were computed from.

A pipeline chains the preprocessing steps and the model into a single object, so that the grid search repeats the whole sequence in a consistent way. The source explains that because tf-idf makes use of these statistical properties of the training data, a pipeline is used to ensure that the results of the grid search are valid. The pipeline keeps the tf-idf step tied to the training data it was built from, rather than letting the search reuse statistics in a way that does not match the comparison it is making.