Sparse Matrices in Text Feature Extraction
The ngram_range=(min_n, max_n) tuple sets the shortest and longest token sequences that CountVectorizer extracts as features, and both bounds are inclusive.
Try it: Text Vectorization: Tokens to Vectors
The text-to-tensor pipeline: standardize, split into tokens, build a frequency-ranked vocabulary with padding and [UNK], map tokens to integer ids, pad or truncate, then one-hot or multi-hot encode and look the ids up in an embedding table.
How it works
- Standardize: lowercase and delete punctuation.
- Split on whitespace into tokens.
- Build the vocabulary: 0 = padding, 1 = [UNK], then words by frequency (ties: first appearance), capped at max tokens.
- Map each token to its id (1 if it did not make the vocabulary), then pad with 0 or truncate to the sequence length.
- Encode: one-hot rows per position or one multi-hot row per text; embed: id k selects row k of the embedding table.
Default run (18 steps): Raw text (59 characters): "The movie was great. The plot was great, the acting wasn't!". A model cannot read strings, so it is turned into numbers step by step. … Embedding lookup: id k selects row k of a fixed 8 x 3 table, giving a (8, 3) dense matrix instead of (8, 8) sparse one-hots. Row 0 (padding) is looked up too; a trained Embedding layer would learn these rows.
Simplified: Bounded ASCII text (up to 200 characters, 40 tokens), whitespace splitting only, and a small FIXED embedding table (a real Embedding layer learns its rows). The multi-hot vector keeps the padding slot (always 0) so its positions line up with the ids.
Loading the simulation…
Why Single Words Lose Word Order
A bag-of-words vectorizer turns text into numbers by counting how often each feature appears in a document. Its default feature is a single token, meaning a single word. That choice throws away word order entirely. The model learns that the words occurred, but not which words sat next to each other. Two sentences made of the same words in a different order look identical to it.
The n-gram range is the setting that lets you keep some of that local order. It tells the vectorizer to build features from short runs of consecutive tokens as well as from single tokens. In this article you will configure the range on CountVectorizer, run it on a small corpus, and read the vocabulary it produces. Each step asks you to predict something first, so you can check your reasoning against the real result.
Reading the ngram_range Tuple
CountVectorizer is the scikit-learn class that builds a vocabulary from a collection of documents and counts features. Its ngram_range parameter takes a tuple written as (min_n, max_n). The tuple sets the shortest and longest token sequences the vectorizer will extract. Both values are inclusive, so the bounds themselves are included in the result.
ngram_range=(min_n, max_n) is the setting on CountVectorizer that sets the shortest and longest token sequences extracted as features. Both min_n and max_n are inclusive.
- (1, 1) means only single tokens, so each feature is one word.
- (2, 2) means only sequences of exactly two tokens, so each feature is one pair of consecutive words.
- (1, 3) means every sequence of one, two, or three tokens, all gathered into one vocabulary.
Predicting the Unigram Vocabulary
Start with the default. Set the range to (1, 1) on the bards_words corpus and print the vocabulary. The source's output lists thirteen features, all single words in alphabetical order. The vectorizer lowercased the text and sorted the vocabulary, which is why 'be' comes first and 'wise' comes last. Each position in this list corresponds to one column of the matrix the vectorizer will produce.
['doth' 'fool' 'he' 'is' 'the' 'think' 'wise']ngram_range=(1, 1) keeps only single tokens.get_feature_names_out()returns the vocabulary in alphabetical order, one entry per column.
Bigrams Keep Local Order
What do you think happens?
Before looking at the bigram output, predict how many features a (2, 2) range produces on the same bards_words corpus. Think about the fact that each pair is a new kind of feature, but a pair repeats less often than a single word.
Reveal answer
14 features, all bigrams.
Pairs are new features, but since each pair is less likely to repeat than a single word, the count only rises from 13 to 14 here, and every feature is now two tokens long.
['doth think' 'fool doth' 'he is' 'is wise' 'the fool' 'think he']- Each feature is two tokens joined by a space, in the order they appear in the text.
- Only adjacent tokens are joined, so 'fool he' or 'the doth' cannot appear.
Notice what the bigram list does not contain. Pairs such as 'fool be' or 'wise the' never appear in the source corpus, because the vectorizer only joins tokens that are adjacent in the original text. The feature 'man knows' exists because 'man' is directly followed by 'knows'. Reading the list this way lets you confirm that the extraction respects word order.
Tracing a Row Back to Its Names
A vectorizer's output is a matrix with one row per document and one column per feature. The dense form is easier to read because it prints every column rather than storing only the nonzero entries. To read a row, pair each position with the feature at the same index. Position 0 holds 'be fool', position 1 holds 'but the', and so on. A count of 1 means that feature appears in that document.
Tracing row 0 of the bigram matrix
Row 0 of the dense bigram matrix has nonzero entries at indices 2, 3, 4, 6, 9, and 11. Which bigrams does this document contain?
- 1Locate the vocabulary
The fourteen bigrams are listed in alphabetical order, and each one sits at the position given by its index.
- 2Map each nonzero index
Index 2 is 'doth think', index 3 is 'fool doth', index 4 is 'he is', index 6 is 'is wise', index 9 is 'the fool', and index 11 is 'think he'.
- 3Check the zero entries
Positions 0 and 1 hold 'be fool' and 'but the' and are zero in row 0, so row 0 does not contain those pairs.
- 4Compare with row 1
Row 1 has nonzero entries at the other eight positions. The two rows never share a nonzero index, which matches the fact that the two phrases have no bigram in common.
Row 0 corresponds to the bigrams 'doth think', 'fool doth', 'he is', 'is wise', 'the fool', and 'think he'. Each nonzero column is one pair that occurs in that document, and the column index is what links the number back to its name.
How Trigrams Grow the Vocabulary
A range such as (1, 3) asks for every sequence of one, two, or three tokens. The vectorizer gathers all three lengths into a single vocabulary, so the feature list mixes single words, pairs, and triples. A trigram is simply a sequence of three consecutive tokens.
The jump from 14 to 39 shows how quickly the feature count grows when longer sequences are added. Each added length contributes its own set of features. Longer sequences usually produce more features and more specific ones, since a three-word phrase is rarer than a single word and therefore a more specific signal.
Common Mistakes With the Range
Common mistake Assuming (2, 2) keeps the single words as well
What it looks likeCreating CountVectorizer(ngram_range=(2, 2)) and then looking for 'fool' in the vocabulary.
Why it failsBoth bounds are inclusive and both equal 2, so only sequences of exactly two tokens are extracted.
Do this insteadUse (1, 2) when you want single words and pairs together.
Common mistake Expecting non-adjacent word pairs in the bigram list
What it looks likeLooking for 'fool be' or 'wise the' in the bigram vocabulary.
Why it failsThe vectorizer joins only tokens that are adjacent in the original text, so pairs that skip over other words are never created.
Do this insteadCheck adjacency in the original sentence before deciding whether a pair should appear.
Common mistake Reading the dense row without the vocabulary
What it looks likeSeeing a count at column 9 and treating the number 9 as if it were a word.
Why it failsA column index has no meaning on its own. It identifies a feature only through the ordered vocabulary.
Do this insteadAlways pair the index with the entry at that position in get_feature_names_out() before interpreting a count.
Common mistake Writing the tuple in the wrong order, as (3, 1) instead of (1, 3), which reverses the shortest and longest bounds
Practice Tracing a Bigram Row
Row 1 of the bigram matrix has nonzero entries at all positions except the six used by row 0. Without looking at the full vocabulary, explain why row 1 cannot contain 'doth think', and state which index in row 0 would tell you that the two rows share no bigram.
Hints
- Row 0 has a nonzero entry at index 2, which is 'doth think'.
- The two rows never share a nonzero index, so any index that is nonzero in row 0 must be zero in row 1.
Set ngram_range to (1, 2) on the sentence 'the fool doth think he is wise' and predict the number of features before running the code. Then check your count against the output.
Hints
- Count the seven unigrams and the six bigrams separately, then add them.
- Single tokens and pairs are both kept because both bounds are inclusive.
Key Takeaways
- The
ngram_range=(min_n, max_n) tuple sets the shortest and longest token sequences CountVectorizer extracts, and both bounds are inclusive. - Bigrams keep local word order, because the vectorizer joins only tokens that are adjacent in the original text.
- On the
bards_wordscorpus, (1, 1) gives 13 features, (2, 2) gives 14, and (1, 3) gives 39, showing that longer sequences add many more features. - A dense row lines up position by position with the vocabulary, so each nonzero count can be traced back to a specific n-gram through its index.