Text Representation with Bag-of-Words

Machine learning algorithms work with numbers, not raw sentences. A sentence such as a line from a poem cannot be fed directly into a learning algorithm, so something has to convert the text into numbers first. This article shows one practical way to do that using the CountVectorizer tool from the scikit-learn library. CountVectorizer converts a collection of text documents into a matrix of token counts, and the source calls that matrix a bag-of-words representation. Each document becomes a row of counts, so the text can be processed numerically.

A learning algorithm needs each input as a set of numbers in a consistent layout. Text does not come in that layout: two documents can be different lengths and use different words. A bag-of-words representation solves this by fixing a list of distinct words across the whole collection, then recording, for each document, how many times each word appears. Every document is then described by the same set of columns, which is what makes the rows comparable.

The source program works on a variable named bards_words, which holds the collection of text documents used throughout the source. The program has four lines, and together they are the whole workflow. Read the code first, then trace what each line is responsible for before you look at the final printed result.

The workflow has two stages. The fit stage examines the documents so the vectorizer knows the words it will count. The transform stage then uses that knowledge to turn the documents into the matrix of counts. Keeping these stages separate in your head makes the program much easier to debug, because a problem usually belongs to one stage or the other.

Read the shape $2 \times 13$ as two rows and thirteen columns. The two rows correspond to the two documents in the collection. The thirteen columns correspond to the thirteen distinct words the vectorizer found across the corpus. Each cell in the matrix therefore belongs to exactly one document (its row) and exactly one vocabulary term (its column).

The repr line gives the shape, but not the contents. It also tells you the matrix is sparse, which means most cells are expected to be zero for typical text. Because of that, the object stores only its nonzero entries rather than writing out every zero. The printed line therefore confirms the dimensions of the result without showing you which words occurred in which document.

Word order is also absent from this representation. Each row records how often each word occurs, not where it sits in the sentence. Two documents that contain the same words in a different order would therefore produce identical count rows. This follows directly from what a bag of words is: a collection of counts with the arrangement discarded.

Suppose you run the same program and the printed line reports a different shape from the one you predicted. Do not guess at the cause. Start at the top of the program and, at each line, ask what that line was supposed to produce. Then compare that expectation with what the next line receives. The first line where the expectation and the reality disagree is the place to look.