Word Counts as Features for Text Classification

Imagine you are building a classifier that reads movie reviews and decides whether each one is positive or negative. A model cannot work directly with sentences, so the first job is to turn each review into numbers. A simple way to do this is to count how many times each word appears in each review. Those counts become the model's input features.

Counting is a reasonable starting point, but it treats every word as equally informative. Common words that show up in almost every review add as much weight as a word that appears in only a few reviews and therefore says something specific about those reviews. This article asks two questions: what happens when the counts are rescaled with tf-idf before the model sees them, and how you can check whether that rescaling is actually helping.

Follow one review through the process. It starts as a string of text. The vectorizer breaks the text into words and assigns each word a numeric value for that review. Those values are the features the classifier sees. In the setup the source describes, the vectorizer is one step inside a pipeline, so the same transformation happens automatically whenever new text goes through the pipeline.

Notice where the labels enter. The text goes into the vectorizer first, and the labels only reach the process at the classifier. The vectorizer produces its features without them. Keep this ordering in mind, because it becomes important when you try to interpret which words tf-idf ranks highly.

Plain word counts are easy to compute and easy to explain, but they give every word the same kind of influence. A word that appears in almost every review contributes its count just as strongly as a word that appears only in a handful of reviews. Yet the second kind of word is the one that helps separate one review from another. Filler that shows up everywhere adds noise rather than distinction.

The source reports that switching from plain word counts to tf-idf gives a noticeable improvement in model performance, and that the evidence is a cross-validation score that goes up when tf-idf replaces the raw counts. The rest of this article looks at what that improvement does and does not tell you.

The core idea of tf-idf is that the weight of a word depends on two things at once: how often it appears in a particular document, and how widely it is spread across the whole collection. The source describes the effect in practical terms: words spread across nearly every document carry less weight, while words concentrated in a smaller set of documents carry more. The same word can therefore count for different amounts depending on how it is distributed.

When the vectorizer processes a collection of reviews, the result is a matrix of features. Each row corresponds to one document, and each column corresponds to one vocabulary word. A cell records the value for that word in that document. Because each word has a fixed column position, you can always trace a feature back to the word it represents.

This mapping matters when you inspect results. Once you know which column is which word, a high value in a column tells you which vocabulary word that document relied on. The vectorizer holds the vocabulary that defines these positions, which is one reason you need the vectorizer object itself and not just the matrix it produced.

After the score improves, the natural next step is to inspect which words tf-idf ranks as most important. The source highlights a caution here: tf-idf is a purely unsupervised technique. It never looks at the labels when it computes its weights. It is fit on the text alone. The labels are only used later, when the classifier is trained and evaluated.

This is why inspection is worth doing carefully. If you look at the top tf-idf words and see distinctive vocabulary, that tells you something about how the collection is spread out. Whether those words line up with the classes is a separate question, and the vectorizer never answered it because it never saw the labels.

A cross-validation score is a metric used to evaluate model performance. The source reports a best cross-validation score of 0.89 for the tf-idf version of the classifier, and a higher score in this comparison is what signals the improvement over word counts. The score summarizes how well the model did on held-out portions of the data.

Cross-validation works by splitting the data into folds, scoring the model on each held-out fold, and then summarizing those per-fold results. The result is a single number that reflects performance across the folds rather than on one lucky or unlucky split.

To inspect the tf-idf words, you need the vectorizer itself, not the whole pipeline. The source's first step for this inspection is to extract the TfidfVectorizer from the fitted pipeline. The vectorizer is one step among the pipeline's steps, alongside the classifier step, so you reach it by navigating from the pipeline object to the step that holds it.

Once you have the vectorizer object, you can look at the features it produced and see which words received the highest weights. Keep the two roles separate in your head: the vectorizer is the piece that learned weights from text, and the classifier is the piece that used those weights together with the labels.