Bag-of-Words Features

A bag-of-words model counts individual words and treats each count as a separate piece of evidence. Consider the words 'not' and 'good'. A model that counts only single words treats them as two unrelated signals, even though the phrase 'not good' means something very different from either word alone. The meaning lives in the order and the neighborhood of the words, and single-word counts throw that information away.

An n-gram feature offers one way to keep some of that local order. Instead of counting only individual items, the model also counts short runs of neighboring items, so the feature itself carries a small amount of context. This article follows one kind of n-gram, the trigram, through three steps: how the sequences are formed, how Python selects them, and how to read a visualization of the trigram features a trained model relied on most.

The easiest way to see how these sequences are formed is to slide a window of fixed length across a sentence. For a window of three words, the first trigram starts at the first word, the next trigram starts at the second word, and so on until the window reaches the end. Each position produces exactly one trigram. Changing n changes the length of the window and therefore the sequences it captures: a window of two produces bigrams, a window of three produces trigrams, and a larger window captures longer runs.

In a real model, the feature names come from a vectorizer that has already built a vocabulary of unigrams, bigrams, and trigrams. Selecting trigrams is then a filter: keep only the names whose word count is exactly three and discard the rest. The filter does not change the names themselves; it only decides which ones move forward.

The same logic can be written with a boolean mask over a NumPy array of names, which is how many pipelines express it. The list comprehension above is easier to follow because each step is visible: split, count, compare, keep. Either version returns the same subset.

After a model is trained, one useful view is a plot of only its trigram features, each paired with the coefficient the model assigned to it. The plot keeps the 40 most influential trigram features. Each bar or label pairs a three-word pattern with its coefficient, so you can see which patterns pushed the model's decision one way or the other. Visualizing trigrams separately gives a direct view of how the model uses features that carry local context.

The trigrams highlighted in this kind of plot include phrases such as 'none of the', 'the only good', 'on and on', 'this is one', and 'of the most'. Each phrase carries a small, meaningful pattern. Reading the plot means asking, for each pattern, whether its coefficient pushes the prediction toward one outcome or the other, and how strongly.

The difference between a single word and a trigram is easiest to see side by side. A single word such as 'not' is ambiguous on its own, while the trigram it sits inside says much more about how it is used.

A trigram filter depends on counting words correctly. A wrong comparison value silently changes which features appear, and the code still runs without any error. The example below uses the same feature names as before. The wrong version measures length in characters with len(name), while the correct version measures words with len(name.split()).