Bag-of-Words Representation

A classifier cannot read a movie review the way a person does. It learns from numbers, so a review written as a sentence has to be converted into a numeric form before any learning can happen. The bag-of-words representation is one of the simplest ways to make that conversion. It describes a document by its word occurrences: which words appear and how many times each one appears.

Keep that surprise in mind. The rest of this article follows the representation step by step, so you can see exactly where the order of words is lost and where the counts come from.

Building a bag-of-words representation takes two steps. First, the text is split into tokens, which are the units you choose to count, most often individual words. Second, the distinct tokens that appear across the whole collection are gathered into a vocabulary, and that vocabulary is given a fixed order. When you call fit on a CountVectorizer, both of these things happen: it tokenizes the training text and builds the vocabulary from the unique words those tokens produce.

The learned vocabulary is stored in the vocabulary_ attribute. You read it the same way you would read any attribute of an object. Its keys are words and its values are column indexes. The printed dictionary is not in the order of the sentences. It is the mapping from each word to its position, and that position is what the later counting step relies on.

In the two-sentence example the source uses, the index numbers run from 0 to 12, so the 13 vocabulary words occupy 13 positions. Read in alphabetical order, 'be' is index 0, 'but' is index 1, 'doth' is index 2, and 'wise' is index 12. The vocabulary runs from 'be' to 'wise', and the words that appear in both sentences sit at indexes 3, 9, and 12, which are 'fool', 'the', and 'wise'. Notice that the diagram above shows only the words needed to read those indexes; the vocabulary contains every distinct word.

To produce the bag-of-words data, call transform on the same list of documents. The result is a matrix with one row per document and one column per vocabulary word. Each cell holds a count: how often that column's word appears in that row's document. Every document ends up with a vector of the same length, because every document is measured against the same vocabulary.

The same counting procedure can be written directly in Python. The sketch below uses plain whitespace splitting as a stand-in for tokenization, which is enough to show the mechanics. It builds a sorted vocabulary from all documents and then produces one count vector per document.

Each printed list is one document's bag-of-words vector, and the position of each number matches the position of a term in the printed vocabulary. A library tool such as CountVectorizer handles tokenization more carefully, but the representation it produces has this same shape.

The matrix returned by transform is stored as a SciPy sparse matrix, which keeps only the nonzero entries. This matters because most documents use only a small part of the vocabulary, so most cells in the grid are zero. Storing every zero would waste memory.

The summary that repr prints for the transform result reports three things: the shape, which is 2 rows by 13 columns; the element type, which is numpy.int64; and the storage format, which is a sparse matrix in Compressed Sparse Row form holding 16 stored elements. A full grid for these two sentences would have 26 cells, so only 16 are nonzero and the other 10 are not stored at all. In a real corpus, such as movie reviews compared against the whole English vocabulary, the share of zeros is far larger.

A sparse matrix hides its zeros, so reading an individual count from it is awkward. The easiest way to check a count is to convert the matrix to a dense NumPy array with toarray. The dense version stores every cell, zeros included, so you can read any count directly. The code below uses the same two documents as before.

When a count looks wrong, the useful question is where your expectation and the output part ways. Three links connect a sentence to a cell in the matrix, and you can check them in order. First, the input text: is the sentence exactly what you think it is? Second, the column that vocabulary_ gives for the word: which index does the word actually have? Third, the value at that column in the dense row: is the number there the one you expected?

The name is a metaphor. Take a review, cut it into individual words, and drop every word into a bag. Once the words are in the bag, you no longer know which came first, which words were attached to which, or how the sentence was built. What you still know is which words appeared and how many times. A document is therefore represented as a bag, or set, of its word occurrences.

The source names three kinds of task where bag-of-words representations are a strong starting point: spam and fraud detection, sentiment analysis, and the movie review classification used as the running example. In each case, the question is which category a document belongs to, and the words it uses are often a useful clue. A message full of promotional wording or a review full of praising words gives a classifier something to work with, even without any understanding of sentence structure.

The same ideas extend to the terms the representation is built from. Tokenization decides what counts as a unit to count, and n-grams are groups of consecutive tokens. Choosing these units is part of the representation work, not a separate afterthought. Because the vocabulary is readable, you can also examine the tokens and features a model depends on. The source encourages this kind of introspection for text applications, both supervised and unsupervised, rather than treating the model as a black box.