Sparse Matrix Representation

This article follows a sentiment analysis task: deciding whether a movie review expresses a positive or negative opinion. The reviews come from the IMDb dataset and were loaded earlier into two lists of strings, one for training and one for testing. Each string is an entire review. A computer cannot work with the words of a review directly, so the bag-of-words process gives each review a numeric form based on which words occur in it and how often.

The scale is what changes compared with the small examples earlier in the chapter. The training list holds 25000 reviews, so the numeric representation has to handle a large collection. The result turns out to be stored in a special format designed for matrices that are mostly zeros. Understanding why that format exists, and how to read what it prints, is the goal of this article.

A sparse matrix representation stores only the non-zero elements of a matrix rather than every cell. The format is meant for large matrices that contain many zero elements. A dense alternative would keep a slot for every cell. In the review-by-word setting, that means a slot for every word in every review, including the many words that never appear in that review.

The diagram below takes one review and lays out its row cell by cell. Each cell is a vocabulary word. The count for that word in this review is shown beneath it. Cells with a count above zero are the ones a sparse format keeps. The zero cells are the empty space a dense layout would still pay for.

The cells marked as changed are the stored non-zero entries. The zero cell for bad is the kind of entry a sparse format does not need to keep. In a four-word toy vocabulary the saving is small. In a real collection with tens of thousands of vocabulary words, most word-review combinations are zero, so the saving becomes large.

The work happens in two separate steps. The fit step is applied to the training text and learns which tokens exist in the collection. It builds a vocabulary and remembers it. The transform step then uses that learned vocabulary to convert text into counts, producing one row per review. The two steps have different jobs, and the order between them matters.

In the course source, the training reviews pass through both steps in one line, and the result is stored in a variable named X_train. The name X is the usual convention for the matrix of inputs a model learns from. The sentiment labels, which say whether each review is positive or negative, are a separate target and are not part of this step. The diagram below shows the messages passed between the stages.

The source prints the matrix using a representation string with three pieces of information: the dimensions of the matrix, the data type of its values, and the storage format. For the training matrix the dimensions read 25000x74849. The rows match the 25000 training reviews, so each row represents one review. The columns equal the size of the vocabulary that fit learned from the training text.

The values have the type numpy.int64. This fits counts, because a count is a whole number. The same printout then reports that the matrix contains 3431196 stored elements in Compressed Sparse Row format. The stored element count is not the number of rows or columns and should not be read as the size of the matrix. It counts only the entries that the sparse format keeps.

The stored fraction is about 0.18 percent of all cells. Nearly every word-review combination is zero, because most words never appear in most reviews. A dense matrix would store all 1871225000 cells, and almost all of that memory would hold zeros. The sparse format keeps only the 3431196 entries that carry counts.

The full training matrix is too large to inspect by hand, so this worked example runs the same pipeline on a two-review corpus. The import line is not shown in the source excerpt, so it is added here. Read the program line by line and predict the printed matrix before checking the output.

Before reading the output, predict its shape. The corpus has two reviews, so there should be two rows. The distinct words are good, movie, plot, and bad, so there should be four columns. The shape should therefore be 2 by 4. The printed matrix confirms it: row one counts good twice, movie once, and plot once, while bad is zero. Row two counts bad once and movie once.

A less obvious mistake is fitting on one text and transforming a different one, then expecting the columns to line up. Transform uses the vocabulary that fit learned. If a second fit were run on different text, the vocabulary would change and the columns would no longer mean the same words. This is a general consequence of how the two steps depend on each other, applied here to the two-review example.

When the output of a bag-of-words pipeline does not match expectations, the source suggests a order of checks. Work through them in sequence, because the first check that fails usually points to the step where the output first diverged from what was expected.