Applying Bag-of-Words to Text Data

Machine learning algorithms work with numbers, not raw sentences. A sentence such as a line of poetry cannot be fed directly into a learning algorithm, so something has to translate it into a numeric form first. This article follows one practical bridge between the two: the CountVectorizer tool from the scikit-learn library. It converts a collection of text documents into a matrix of token counts, and the source calls that matrix a bag-of-words representation. Each document becomes a row of counts, so the text can be processed numerically.

The method here is to run the code rather than read about it first. Before you look at the final printed line of each program, try to say in your own words what each line is responsible for. Then check your guess against what the line actually does.

The program works on a variable named bards_words, which holds the collection of text documents used in the source. The source imports CountVectorizer from sklearn.feature_extraction.text, creates an instance named vect, fits it to the documents, and then transforms the same documents into a bag-of-words result. The four lines below are the whole workflow, so each one is worth naming as you trace it.

The program has two distinct stages. The fit call examines the documents so the vectorizer knows the words it will count. The transform call then uses that knowledge to turn the documents into the matrix of counts. The fit call is not the call that produces the result. It only prepares the vectorizer. Keeping the two stages separate in your head makes the program much easier to debug, because a problem usually belongs to one stage or the other.

Notice that transform receives the documents and the vocabulary learned by fit. This explains where problems tend to hide. If the vocabulary was learned from the wrong documents, the columns are wrong. If the counts were produced from the wrong documents, the rows are wrong. Those are two different questions, and asking them separately narrows the search.

The repr of the bag-of-words result reports a sparse matrix with shape $2 \times 13$. Read this as two rows and thirteen columns. The two rows correspond to the two documents in the collection. The thirteen columns correspond to the thirteen distinct words the vectorizer found across the whole corpus. Each column is a word, and each row is a document, so a single cell answers the question: how many times does this word appear in this document?

The source describes this result as sparse. For typical text, most cells are zero, because any one document uses only a small fraction of all the words in the corpus. A sparse object therefore stores only its nonzero entries rather than every cell. The repr line signals this storage choice, but the learner's mental model should still be the full grid of documents by terms.

Before any counting happens, each document must be split into tokens, the individual words that the vectorizer will count. The vectorizer's recorded vocabulary is simply the set of distinct tokens found across the collection, and each distinct token becomes one column. This is why the column count depends on the whole corpus and not on any single document.

Suppose you run the same program and the printed line reports a different shape from the one you predicted. The source gives you enough structure to find the failure point without guessing. Start at the top of the program and ask, at each line, what that line was supposed to produce. Then compare that expectation with what the next line receives. The first line where the expectation and the reality disagree is the place to look.