Building Text Classification Pipelines

Because the search wraps the whole pipeline, every candidate setting is judged by the final classifier's cross-validated score.

Programming

A Question About Word Pairs

Imagine you want a classifier that reads movie reviews and decides whether each one is positive or negative. The simplest design looks only at single words. Should the classifier look only at single words, or does adding word pairs and word triples make its sentiment predictions better? This article follows one experiment that answers that question on the IMDb movie review data, using a TfidfVectorizer and a grid search to find the best n-gram range.

The question is simple to state, but the answer depends on the machinery underneath. A pipeline turns text into TF-IDF features and then fits a logistic regression. A parameter grid lists the values the search may try for two settings, the regularization strength C and the n-gram range. GridSearchCV then tries every combination, scores each one with cross-validation on the training text, and keeps the winner. As you read, treat the code as a sequence of decisions: which values get tried, in what order, and what the search returns at the end.

Counting the Work Before the Search Runs

Before looking at how many times the search fits a model, make a prediction from the grid alone. In the source's experiment, the C list has six values, the ngram_range list has three tuples, and the search uses five folds. Each fold is one train-and-score cycle for one combination of settings. Commit to a number before you continue, because that number tells you how much work the search is about to do.

What do you think happens?

With 6 values of C, 3 ngram_range tuples, and 5 folds, how many times does GridSearchCV fit the pipeline?

Reveal answer

Answer:

90 fits, because $6 \times 3 \times 5 = 90$

GridSearchCV fits the pipeline once for every combination of settings in each cross-validation fold. The total is the product of the grid sizes and the number of folds.

How the C values, n-gram ranges, and CV folds multiply together into the total number of fits, and in what nested order the search visits each combination.

The diagram shows the nesting. The search picks one C value, then walks through all three n-gram ranges for that C, and within each pairing it runs every fold. Each innermost step is one fit and one score. Multiplying the three counts gives the total work: $6 \times 3 \times 5 = 90$ fits. The source warns that this is why the search takes a long time, so the count is worth knowing before you press run.

What the Search Actually Tunes

The search does not tune a vectorizer or a classifier on its own. It tunes a pipeline. The source builds one with make_pipeline, passing a TfidfVectorizer with min_df=5 and a LogisticRegression. Text enters the first step, the vectorizer turns it into a matrix of TF-IDF features, and the second step fits a classifier on that matrix.

Because the search wraps the whole pipeline, every candidate setting is judged by the final classifier's cross-validated score. A vectorizer setting is never judged by how its output looks in isolation. It is judged by how well the classifier that depends on it performs on held-out data.

For each CV fold, which rows go to the vectorizer fit, which go to the classifier, and where the held-out validation data enters the flow.

Reading N-Gram Ranges

Definition

An n-gram is a contiguous sequence of n items from a text. Unigrams are single words, bigrams are pairs of neighboring words, and trigrams are triples of neighboring words.

The ngram_range value is a tuple of two numbers, a lower bound and an upper bound on the length of the word sequences to extract. The tuple (1, 1) keeps single words only. The tuple (1, 2) keeps single words and two-word sequences. The tuple (1, 3) keeps single words, two-word sequences, and three-word sequences. Each tuple is one candidate the search will try.

Which n-gram types are contained in the feature set produced by ngram_range=(1, 3), and how they relate to one another.

The range is inclusive. A setting of (1, 3) does not replace single words with pairs and triples. It keeps all three kinds together, so single words remain in the feature set alongside the longer sequences. This matters for the experiment's motivation: the phrase not good and the word good look different to a classifier only when the pair is kept as a feature. That is why the search compares ranges that include pairs and triples against ranges that do not.

Writing the Parameter Grid Keys

The keys of the param_grid dictionary are not arbitrary labels. Each name has two parts joined by a double underscore: the name of a pipeline step, then the name of a parameter that step owns. The source writes logisticregression__C, which means the C parameter of the LogisticRegression step, and tfidfvectorizer__ngram_range, which means the ngram_range parameter of the TfidfVectorizer step. The search uses these names to route each candidate value to the correct object inside the pipeline.

In the key tfidfvectorizer__ngram_range, which part names the pipeline step, which part names the parameter, and how the double underscore routes the value to that step.

python

from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV

pipe = make_pipeline(TfidfVectorizer(min_df=5), LogisticRegression())

param_grid = {
    'logisticregression__C': [0.01, 0.1, 1, 10, 100, 1000],
    'tfidfvectorizer__ngram_range': [(1, 1), (1, 2), (1, 3)],
}

grid = GridSearchCV(pipe, param_grid, cv=5)
grid.fit(X_train, y_train)

from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV

pipe = make_pipeline(TfidfVectorizer(min_df=5), LogisticRegression())

param_grid = {
    'logisticregression__C': [0.01, 0.1, 1, 10, 100, 1000],
    'tfidfvectorizer__ngram_range': [(1, 1), (1, 2), (1, 3)],
}

grid = GridSearchCV(pipe, param_grid, cv=5)
grid.fit(X_train, y_train)

Notes on the code

make_pipeline builds the two-step pipeline; the steps get default names from their class names
min_df=5 is fixed for the vectorizer and is not part of the grid
The C list here is illustrative; the source's list has six values
Each key is step name, double underscore, parameter name
cv=5 gives five folds, so the search performs 6 x 3 x 5 fits
grid.fit receives training text and training labels only

Following the Search to Its Result

Now trace the search through the source's run. For each combination, the search cross-validates over the folds, records the mean score, and checks whether combinations remain. When none remain, it reports the best score and the parameters that produced it. In the source, that winning combination had ngram_range (1, 3) and C equal to 100, with a best cross-validation score of 0.91.

Those numbers describe one winning combination. They do not say how much each n-gram setting contributed. The source's heat map plots mean cross-validation accuracy as a function of ngram_range and C. It shows that moving from unigrams to bigrams raises accuracy quite a bit, while adding trigrams gives only a very small further benefit. The source states that adding bigram and trigram features improved performance by a bit more than a percent. The best setting still includes trigrams, so the winner and the largest single gain are not the same step.

Worked example

Reading the Source's Result

The search reports a best score of 0.91 with ngram_range (1, 3) and C = 100. Does that tell you that trigrams caused the improvement?

1
Identify what the number measures
0.91 is the mean cross-validated score of one combination, the winner of the whole search.
2
Identify what the number does not measure
It does not split the score into contributions from unigrams, bigrams, and trigrams.
3
Compare settings that differ in one thing
Going from (1, 1) to (1, 2) changes the bigram inclusion. The heat map shows this step raised accuracy quite a bit.
4
Compare the next step
Going from (1, 2) to (1, 3) changes only the trigram inclusion, and the source reports only a very small further benefit.

Result

Bigrams produced most of the gain, trigrams added a little more, and the best setting happens to include trigrams. The 0.91 score alone cannot separate these contributions.

How accuracy moves as the n-gram range grows, based on the source's description: the big change comes from adding bigrams, while trigrams add only a little.

Mistakes That Make Results Look Wrong

Common mistake
Using a bare parameter name as a grid key
What it looks like
```
{'C': [1, 10, 100]}
```
Why it fails
The key does not say which pipeline step owns the parameter, so the value may not reach the object you meant to tune.
Do this instead
Use the step prefix, as in 'logisticregression__C': [1, 10, 100].
Common mistake
Assuming the best score explains each setting
What it looks like
Reading 0.91 as the score contributed by trigrams
Why it fails
The best score is one number for one combination. It does not measure how much each n-gram setting added.
Do this instead
Compare combinations that differ in only one setting, as the heat map does across ranges.
Common mistake
Forgetting the fold multiplier when estimating run time
What it looks like
Counting 18 fits for 6 C values and 3 ranges
Why it fails
Every combination is fit once per fold, so the count is larger than the grid size alone.
Do this instead
Multiply by the folds: $6 \times 3 \times 5 = 90$ fits.
Common mistake
Judging the vectorizer alone, instead of the pipeline's classifier score, which is the only score the search uses to pick a winner

Locating Where Results Diverge

When a grid search result differs from what you expected, the cause usually sits in one of three places. Check them in this order, so that you find the divergence before you change any code.

1
Check the parameter names. Confirm that each key uses the correct step prefix, such as tfidfvectorizer__ngram_range, so the value reaches the object you meant to tune.
2
Check the data the search sees. The source calls grid.fit with the training text and training labels, so the choice of setting should depend only on training data. Confirm that your call passes only training data.
3
Check the comparison you are making. The best score is one number for one combination. To isolate an effect, compare combinations that differ in only one setting, such as (1, 2) against (1, 3) at the same C.

Predict, Then Check the Search

MEDIUM

A colleague writes param_grid = {'C': [1, 10], 'tfidfvectorizer__ngram_range': [(1, 1), (1, 2), (1, 3)]} and runs GridSearchCV with 5 folds. Before you answer, predict two things: how many fits the search performs, and which part of the grid is most likely to go astray.

Hints

Count the C values, the ngram_range tuples, and the folds, then multiply.
Look at the first key. Does it name a step, and does the prefix match the step name?

Your attempt (not saved)

EASY

Rewrite the first key so it routes correctly, and state the new number of fits.