Feature Extraction with CountVectorizer and TfidfVectorizer
Feature extraction in CountVectorizer and TfidfVectorizer is described as relatively simple, and it can be improved.
Where a Program Starts Reading Text
A person reading a sentence never has to think about where its words begin and end. The spaces, capital letters, and punctuation marks make the boundaries obvious. A program has no such intuition. Before a computer can count anything in a document, it has to agree on what the countable pieces are. It needs an explicit rule that cuts a stream of characters into units.
That cutting step is called tokenization. This article follows one sentence through two different cutting rules and shows that the rule, not the counting, is where much of the work of text feature extraction is decided. The article then explains why the simple approach used by CountVectorizer and TfidfVectorizer has room for improvement.
What do you think happens?
Take the sentence: The cat sat on the mat. If a word is defined as anything between spaces, what would the units be?
Reveal answer
The, cat, sat, on, the, mat.
Splitting on spaces keeps the capital T in The and keeps the period attached to mat, because the period sits between the word and the following space in the raw text only once the sentence ends. Nothing in the rule removes it, so mat. stays a single unit.
Tracing One Sentence Through Two Rules
The following example is an original illustration, not taken from the source. It applies two different rules to the same sentence. The first rule says a word is anything between spaces. The second rule says a word is a run of letters, and it also lowercases the text first. Follow each step and watch which units survive.
Splitting The cat sat on the mat under two rules
Sentence: The cat sat on the mat. Apply Rule A (a word is anything between spaces) and Rule B (lowercase the text, then a word is a run of letters). List the tokens each rule produces.
- 1Apply Rule A
Cut the sentence at every space. The pieces are The, cat, sat, on, the, and mat. with the period still attached to the last piece.
- 2Apply Rule B lowercasing first
Lowercase the whole sentence before cutting. The text becomes the cat sat on the mat. so The is now the and the first letter loses its capital.
- 3Keep only runs of letters
Under Rule B a word is a run of letters, so the period is not part of any word. The last token becomes mat with no period.
- 4Compare the two token lists
Rule A yields six units including The and mat. Rule B yields six units including the and mat, so the same sentence now has different units even though the number of pieces is the same.
Rule A produces The, cat, sat, on, the, mat. while Rule B produces the, cat, sat, on, the, mat. The two lists differ in case and in the period, and any later count will reflect those differences.
Why Tokenization Comes First
The bag-of-words model depends on this first decision. Once the text has been cut into units, the later stages only see those units. Feature extraction never sees the raw characters directly; it only sees what tokenization hands it. This is why the source treats tokenization as a foundational choice rather than a minor housekeeping task.
Read the pipeline from left to right. Tokenization turns characters into units, and feature extraction counts those units. Nothing downstream can recover a distinction that tokenization erased. If the cutting rule is careless, the mistake made at the beginning spreads into every count that comes after it.
Why the Simple Approach Leaves Room for Improvement
The source describes the feature extraction in CountVectorizer and TfidfVectorizer as relatively simple, and notes that much more elaborate methods are possible. The first step that tends to be improved in more sophisticated text-processing applications is tokenization itself.
The reason follows from the earlier trace. A fixed rule about whitespace or letters will not always match what an application needs a word to be. In the example, Rule A keeps mat. with its period and Rule B removes it, and neither rule is universally right. A rule that works well for one kind of text may split or merge units in ways that hurt the next stage.
Second Trace With a New Sentence
Now apply the same idea to a different sentence. The rule has not changed, but the input has. Trace the units yourself before reading the answer.
What do you think happens?
Sentence: Don't stop, keep going. Under the rule that a word is a run of letters after lowercasing, which tokens appear?
Reveal answer
don, t, stop, keep, going
Lowercasing gives don't stop, keep going. A run of letters stops at the apostrophe, so don and t become separate units. The comma and period are not letters, so they do not appear. The contraction is split in a way the rule did not intend, which is exactly the kind of mismatch the source describes.
Mistakes When Reading Token Lists
Common mistake Assuming a word means the same thing to every program
What it looks likeBelieving that The and the are always the same unit regardless of rule.
Why it failsThe source states that the rule defines what counts as a word, so the same text can produce different units under different rules.
Do this insteadAlways state the tokenization rule before comparing two token lists.
Common mistake Thinking the counts are wrong when they simply reflect a different rule
What it looks likeSeeing mat and mat. as separate features and concluding the counter has a bug.
Why it failsThe counter counted exactly the units the rule produced. The difference originates upstream in tokenization.
Do this insteadTrace back to the tokenization rule before debugging the counting step.
Common mistake Treating tokenization as a minor preprocessing detail
Why it failsThe bag-of-words model only sees what tokenization hands it, so a careless rule affects every later count.
Do this insteadGive the tokenization rule the same attention you give the counting method.
Practice Tracing a Rule
Take the sentence: Data is fun! Under Rule A (anything between spaces) list the tokens. Then under Rule B (lowercase the text, then a word is a run of letters) list the tokens. Which tokens differ between the two lists, and why?
Hints
- Under Rule A, the exclamation mark stays attached to fun.
- Under Rule B, the text is lowercased first and the exclamation mark is not a letter.
- Compare the first token and the last token of each list.
Key Takeaways From the Tokenization Trace
- 1Tokenization is the rule that cuts a stream of characters into the units a program will count.
- 2The same sentence can produce different units depending on the rule, and the rule shapes every later count.
- 3Tokenization is the first step of the bag-of-words model, and feature extraction only sees the units it receives.
- 4The simple feature extraction in CountVectorizer and TfidfVectorizer is relatively simple, and tokenization is often the first stage improved in more sophisticated applications.
At a glance
- Tokenization is the rule that cuts a stream of characters into the units a program counts.
- The same sentence can produce different units under different rules, so the rule shapes every later count.
- Tokenization is the first step of the bag-of-words model, and feature extraction only sees what tokenization hands it.
- The simple feature extraction in CountVectorizer and TfidfVectorizer leaves room for improvement, and tokenization is often the first stage improved in more sophisticated applications.