Sparse Matrices for Text Data

Up to now, the bag-of-words process has been shown on small, hand-made examples. This article moves to a realistic setting: sentiment analysis of movie reviews from the IMDb data. The training and test reviews have already been loaded into lists of strings, and the same counting procedure is now applied to thousands of reviews at once.

When you run a vectorizer on real text, the output can look surprising. The most useful skill at this stage is knowing where a surprise comes from. Each stage of the process has one specific job, so a mismatch between what you expected and what you see always points back to one stage. This article builds that habit, starting with the vocabulary itself.

A very common assumption at this point is that the vocabulary is a curated list of meaningful words. Under this picture, the vectorizer reads a review, recognizes the important words, and skips the rest. A number such as a year would never become a feature, because it carries no sentiment.

The mechanical reality is different. The vocabulary is built by scanning the training reviews and recording every token that appears. Whether a token is meaningful, a number, a misspelling, or a variant of another word, the process treats it the same way: it becomes a feature. The vectorizer does not evaluate meaning. It counts occurrences.

The process has two distinct calls, and keeping them apart is the key to reading the output. The fit call scans the training reviews and builds the vocabulary. It determines which words exist and assigns each one a feature position. The transform call then takes the reviews and produces the count matrix, where each entry records how often a feature occurs in a given review.

When you trace these two lines, pay attention to which object each call returns and what it is responsible for. The vectorizer object holds the vocabulary after fitting, and the matrix comes out of the transform step. If a result is wrong, you can ask which of these two calls introduced it. A wrong vocabulary points to fitting; a wrong set of counts for a correct vocabulary points to transforming.

The shape of the resulting training matrix is 25,000 by 74,849. The number of rows corresponds to the 25,000 training reviews, and the number of columns, 74,849, matches the number of vocabulary entries. This is the first check to make on any output: the column count should equal the vocabulary size, and the row count should equal the number of documents passed in.

The bag-of-words matrix is stored as a SciPy sparse matrix, specifically in Compressed Sparse Row format. A sparse matrix stores only the non-zero elements, which conserves memory when most entries are zero. Its printed description also reports how many stored elements it contains.

The printed description of X_train reports 3,431,196 stored elements. The full grid has 25,000 times 74,849 positions, which is $1{,}871{,}225{,}000$ cells. The ratio is $\frac{3{,}431{,}196}{1{,}871{,}225{,}000} \approx 0.0018$, so only about 0.18 percent of the cells hold stored values. Almost every cell is a count of zero, because almost every word in the vocabulary is absent from almost every review.

The diagram is a teaching simplification and not a literal memory layout. Its entries illustrate the principle: a stored element records a word that is present in a review and how often it occurs. Words that do not occur in the review simply have no entry, which is why the matrix can be large without being expensive to hold.

The vectorizer exposes the vocabulary through get_feature_names, which returns a list in which each entry corresponds to one feature. The list index is the feature's column position in the matrix. Because it is an ordinary Python list, you can take slices of it to inspect any region of the vocabulary without printing all 74,849 entries.