Pipelines in scikit-learn for Text Models

Imagine you are building a classifier that reads movie reviews and decides whether each one is positive or negative. A model cannot work directly with sentences, so the first job is turning each review into numbers. A simple way to do this is to count how many times each word appears in each review. Those counts become the model's input features.

Counting is a reasonable starting point, but it treats every word as equally informative. Common words that show up in almost every review add as much weight as a word that appears in only a few reviews and therefore says something specific about them. This article follows one review through the process, explains how tf-idf rescales the raw counts, and shows how to check whether that rescaling is actually helping.

Follow one review through the process. It starts as a string of text. The vectorizer breaks the text into words and assigns each word a numeric value for that review. Those values are the features the classifier sees. In the setup described by the source, the vectorizer is one step inside a pipeline, so the same transformation happens automatically whenever new text goes through the pipeline.

The diagram shows the chain as a sequence. Raw review text enters the vectorizer, the vectorizer produces tf-idf features, the model step learns from those features, and the resulting cross-validation score is what you compare across experiments. The vectorizer is the only step that turns text into numbers, which is why it is the object you will later pull out for inspection.

The practical effect of tf-idf is that words spread across nearly every document carry less weight, while words concentrated in a smaller set of documents carry more. A reviewer's distinctive vocabulary is more likely to earn a high score than filler that appears in every review. This is the sense in which tf-idf rescales the raw counts: the same word can count for different amounts depending on how it is distributed across the collection.

The source reports that the switch from plain word counts to tf-idf gives a noticeable improvement in model performance. The evidence is the cross-validation score, which goes up when tf-idf replaces the raw counts. A cross-validation score is a metric used to evaluate model performance. The source reports a best cross-validation score of 0.89 for the tf-idf version of the classifier, and a higher score in this comparison is what signals the improvement over word counts.

Read the number as a summary of how well the model did on held-out portions of the data, not as a verdict on whether the features are meaningful. Two runs can produce different scores for reasons that have nothing to do with the words themselves. A score is therefore most useful when you compare setups that differ in one deliberate way, such as counts versus tf-idf.

After the score improves, the natural next step is to inspect which words tf-idf ranks as most important. The source gives a caution here: tf-idf is a purely unsupervised technique. It never looks at the labels when it computes its weights. The labels are only used later, when the classifier is trained and evaluated.

To inspect the tf-idf words, you need the vectorizer itself, not the whole pipeline. The source's first step for this inspection is to extract the TfidfVectorizer from the pipeline. Once you have that object, you can look at the features it produced and see which words received the highest weights.