Unigrams and Single-Token Features
The range of token sequences that a vectorizer considers is controlled by the ngram_range parameter of CountVectorizer or TfidfVectorizer.
Try it: Text Vectorization: Tokens to Vectors
The text-to-tensor pipeline: standardize, split into tokens, build a frequency-ranked vocabulary with padding and [UNK], map tokens to integer ids, pad or truncate, then one-hot or multi-hot encode and look the ids up in an embedding table.
How it works
- Standardize: lowercase and delete punctuation.
- Split on whitespace into tokens.
- Build the vocabulary: 0 = padding, 1 = [UNK], then words by frequency (ties: first appearance), capped at max tokens.
- Map each token to its id (1 if it did not make the vocabulary), then pad with 0 or truncate to the sequence length.
- Encode: one-hot rows per position or one multi-hot row per text; embed: id k selects row k of the embedding table.
Default run (18 steps): Raw text (59 characters): "The movie was great. The plot was great, the acting wasn't!". A model cannot read strings, so it is turned into numbers step by step. … Embedding lookup: id k selects row k of a fixed 8 x 3 table, giving a (8, 3) dense matrix instead of (8, 8) sparse one-hots. Row 0 (padding) is looked up too; a trained Embedding layer would learn these rows.
Simplified: Bounded ASCII text (up to 200 characters, 40 tokens), whitespace splitting only, and a small FIXED embedding table (a real Embedding layer learns its rows). The multi-hot vector keeps the padding slot (always 0) so its positions line up with the ids.
Loading the simulation…
Two Sentences, One Count
A common way to turn text into numbers is the bag-of-words representation. It counts how often each token appears in a document and records those counts. The representation is convenient, but it throws away one important piece of information: where each token sat in the text. Once the positions are gone, two different sentences can end up with exactly the same numbers.
What do you think happens?
Consider the sentence 'it is not good'. If the vectorizer uses the default settings, how many features does it create for this sentence?
Reveal answer
4
The default setting creates one feature for each single token. The sentence has four tokens: it, is, not, and good. Pairs of tokens are not created unless they are requested explicitly.
What a Bag of Words Cannot See
The source uses two strings that differ only in the placement of the words 'good' and 'bad' around the word 'not'. The first is 'it's bad, not good at all'. The second is 'it's good, not bad at all'. Both strings contain the same tokens with the same counts: one each of it's, bad, not, good, at, and all. A plain bag-of-words representation therefore gives both strings identical numbers, even though their meanings are inverted.
The problem is not specific to the word 'not'. Placing 'not' in front of a word is only one extreme example of how context changes meaning. It is the clearest case, though, because the same tokens appear in both strings and only their arrangement differs. Any representation that counts single tokens alone has this blind spot.
Adjacent Tokens as Features
The fix keeps the same counting idea but widens what gets counted. Instead of counting only single tokens, the representation also counts pairs or triplets of tokens that sit next to each other. Because each counted unit now contains neighbouring words, the representation keeps some of the local context that single-token counts discarded.
An n-gram is a sequence of adjacent tokens. A unigram is a sequence of one token, a bigram is a sequence of two adjacent tokens, and a trigram is a sequence of three adjacent tokens. The prefix n stands for the length of the sequence.
The word 'adjacent' carries the weight of this definition. A bigram is not any two words from the sentence. It is two words that sit next to each other in the original order. In the sentence 'it is not good', the bigrams are 'it is', 'is not', and 'not good'. The pair 'it not' is not a bigram of this sentence, because 'it' and 'not' are not neighbours.
One way to picture this is a window that slides across the token sequence. A window of size two covers two neighbouring tokens at each position. Each position produces one bigram, and the window moves one token to the right before it reads the next pair.
Reading the ngram_range Tuple
The range of token sequences that a vectorizer considers is controlled by the ngram_range parameter of CountVectorizer or TfidfVectorizer. Its value is a tuple of two numbers, written as ngram_range = (min_length, max_length). The first number is the shortest sequence length to include, and the second is the longest.
Both numbers are inclusive. A tuple of (1, 2) asks for every sequence that is at least one token long and at most two tokens long, which means all unigrams and all bigrams. A tuple of (2, 2) asks for bigrams only, because the minimum and the maximum are both two. The tuple describes a window of lengths, not a position in the sentence.
The default setting covers only one length. It creates one feature per sequence that is at least one token long and at most one token long, which is exactly one token long. That is why the default representation consists of single tokens, the unigrams. Pairs of tokens must be requested explicitly by changing the tuple.
Tracing a Range on 'it is not good'
Predicting the features for ngram_range (2, 3)
The sentence is 'it is not good'. The tokens are it, is, not, and good. Predict every feature produced by ngram_range = (2, 3), and say how many features that is.
- 1Identify the allowed lengths
The minimum is 2 and the maximum is 3. Both bounds are inclusive, so lengths 2 and 3 are produced. Unigrams of length 1 are excluded.
- 2List the bigrams
Slide a window of size two across the four tokens. The adjacent pairs are 'it is', 'is not', and 'not good'. That gives three bigrams.
- 3List the trigrams
Slide a window of size three. The adjacent triplets are 'it is not' and 'is not good'. That gives two trigrams.
- 4Count the features
The vocabulary contains only the bigrams and trigrams, so it has three plus two, which is five features. No single-token column such as 'good' by itself appears.
Five features: it is, is not, not good, it is not, and is not good. The default (1, 1) would give four features, and (1, 2) would give seven.
Notice what changed when the range moved. The minimum rose from one to two, so the single-token columns disappeared. The maximum rose from two to three, so the trigram columns appeared. The combined range (1, 2) keeps every feature from both unigram-only and bigram-only settings, which is why it is the widest vocabulary of those three.
Setting ngram_range in Code
The following sketch shows where the tuple goes. It fits a CountVectorizer on the single sentence from the worked example, asking for unigrams and bigrams together. Compare it with the default by changing only the ngram_range argument.
CountVectorizer()with no arguments uses the defaultngram_range, which produces unigrams only.ngram_range=(1, 2) asks for every sequence from one to two tokens long, inclusive.get_feature_names_out()returns the vocabulary, so its length is the number of columns each setting creates.
Working through the counts by hand, the default setting creates four columns, one for each token. The (1, 2) setting creates seven columns: those four unigrams plus the three bigrams it is, is not, and not good. The number of columns changes exactly because the range changed, which is the point the source asks you to watch for in the output.
Mistakes Learners Make with ngram_range
The tuple is easy to misread, and the errors below are common when first working with n-gram features.
Common mistake Assuming the default setting already includes bigrams
What it looks likeUsing CountVectorizer() on 'it is not good' and expecting the phrase 'not good' as a column.
Why it failsThe default range is one token long at most and one token long at least, so only single tokens are counted.
Do this insteadRequest the longer sequences explicitly, for example with ngram_range=(1, 2).
Common mistake Treating a bigram as any two words from the sentence
What it looks likeCounting 'it not' as a bigram of 'it is not good'.
Why it failsA bigram must be two tokens that sit next to each other in the original order. 'it' and 'not' are not neighbours in this sentence.
Do this insteadSlide a window across the tokens in order and take only the adjacent pairs: 'it is', 'is not', and 'not good'.
Common mistake Reading (2, 2) as a position in the sentence
Why it failsThe tuple describes a window of allowed lengths, not where in the sentence the features come from.
Do this insteadRead (2, 2) as 'only sequences of length two', which means bigrams only.
Common mistake Thinking the upper bound is exclusive
Why it failsBoth numbers are inclusive, so (1, 2) includes length two.
Do this insteadInclude both endpoints when predicting the lengths produced.
Predicting Features for a New Range
Using the sentence 'it is not good', predict how many features ngram_range = (2, 3) produces, and list them. Then check your answer against the worked example earlier in this article.
Hints
- Start by deciding which lengths the tuple includes, remembering that both bounds are inclusive.
- Count the adjacent pairs first, then the adjacent triplets, and add the two totals.
- Single tokens such as 'good' on their own are not included when the minimum is two.
Key Takeaways
- A bag-of-words representation counts tokens and discards their order, so sentences with the same words in a different arrangement can look identical even when their meanings are opposite.
- An n-gram is a sequence of adjacent tokens: unigrams are single tokens, bigrams are pairs, and trigrams are triplets. Adjacency, not mere co-occurrence, defines them.
- The
ngram_rangetuple (min_length,max_length) sets an inclusive window of sequence lengths, and the default (1, 1) produces only unigrams. - Changing the lower or upper bound changes which lengths are included, which in turn changes the number of columns in the vocabulary.