Text Classification with Bag-of-Words Features

Imagine you need to sort thousands of short messages into two groups, such as spam and not spam, and you need the result to come back almost immediately. Many text problems look like this: the data is large, the features are numerous, and speed matters as much as the last fraction of accuracy. Naive Bayes classifiers are a family of methods built for this situation. They resemble the linear models you have already studied, and their decision rules take a similar shape, but they are built around a different idea of how to learn.

The trade-off is worth stating up front. Naive Bayes models are often slightly less accurate at generalizing than linear classifiers such as LogisticRegression and LinearSVC. What they give up in accuracy, they recover in speed: they are noticeably faster to train than those linear methods and very fast to make predictions. In this article you will follow one small dataset through the whole process, watching the model's internal state change at each step, and then decide which Naive Bayes variant fits a given kind of input.

A classifier cannot read words directly; it needs numbers. The bag-of-words idea solves this by fixing a vocabulary, a list of words the model has agreed to track, and then describing each sentence by which vocabulary words it contains. Every vocabulary word occupies a fixed column position. Position 0 always refers to the same word, position 1 always refers to another word, and so on. The order of words in the sentence is thrown away, which is why the representation is called a bag: only the contents matter, not the arrangement.

Read the diagram as a lookup table with fixed slots. The word free always lives in slot 0, so a value of 1 there means free appeared in the sentence and a value of 0 means it did not. Because every sentence is described against the same vocabulary in the same order, every row in a dataset has the same length and each column can be compared across rows. This is what allows a learning algorithm to ask, for example, how often column 2 is 1 among messages of one class.

Depending on the variant you use, a slot can hold a binary value (the word is present or absent) or an integer count (how many times the word appears). The difference matters later, when you choose a Naive Bayes model, because each variant is designed around one of these kinds of values.

Before any theory, start with a small program. The dataset below has four data points and four binary features each. Each data point belongs to one of two classes, A or B. The program counts how many times each feature is set to 1 within each class. Those counts are the raw material a Naive Bayes model works with. Read the code once and notice which variable holds the features and which holds the labels.

Notice what happened to the labels. They were never used to change the features; they only decided which running total each row was added to. This is the essence of the counting step. The model reads each feature separately and gathers, for every class, how often that feature was on. Nothing about the features is adjusted together with the others.

A Naive Bayes model learns simple per-class statistics for each feature. It looks at one feature at a time and records, for every class, how that feature behaves. The per-class counts you traced above are exactly this kind of statistic. When the model is asked to classify something new, it combines the per-feature evidence from each class to decide which class fits best.

The word naive refers to this simplifying choice: the model treats features separately rather than modeling how they relate to each other. A linear model, by contrast, optimizes all of its feature weights together. Because each feature in a Naive Bayes model is handled independently and the work per feature is cheap, the whole training process stays short.

Prediction is also quick because the model has already stored its statistics. Classifying a new row means looking up the stored evidence for each feature that the row contains and combining them. No large search or joint optimization is needed at prediction time, which is why Naive Bayes models are described as very fast to predict as well as to train.

scikit-learn implements three Naive Bayes classifiers. They share the same basic idea and differ mainly in the type of data they assume each feature contains. Choosing the right one starts with asking what a single feature value means in your data.

The three variants are compared side by side below in terms of the data each one expects.

GaussianNB is meant for continuous data of any kind. BernoulliNB assumes each feature is binary, meaning it is either present or absent. MultinomialNB assumes count data, where each feature is an integer count such as how often a particular word appears in a sentence. Both BernoulliNB and MultinomialNB are used most often for classifying text. Applied to the tiny dataset from earlier, the 0s and 1s are binary, so BernoulliNB matches that data directly; if the same slots instead held how many times each word occurred, MultinomialNB would be the matching choice.

When two text variants both seem plausible, the source notes that MultinomialNB usually performs better than BernoulliNB, particularly on datasets with a relatively large number of nonzero features, such as large documents.

MultinomialNB and BernoulliNB each have one main tuning parameter, called alpha, which controls model complexity. The mechanism is smoothing. The algorithm adds alpha virtual data points that have positive values for all features, which softens the statistics it collected from the real data. A larger alpha means more smoothing and a less complex model.

Smoothing matters most for features that never occur with a class. In the tiny dataset, feature 3 has a count of 0 in class A. Without smoothing, the statistics for that feature would claim that feature 3 is impossible in class A, which is a very strong conclusion from only two rows. Adding alpha virtual points gives the zero count a small positive value, so one missing example does not make the model treat the feature as ruled out.

Naive Bayes models share many strengths and weaknesses with linear models. They train and predict very quickly, their training procedure is easy to follow, and they handle high-dimensional sparse data well. They are also relatively robust to their parameters. Because of this, they serve as strong baseline models and are often used on very large datasets where training even a linear model might take too long.

Naive Bayes models are simple to run, which makes it easy to skip the questions that matter most. The mistakes below are tied directly to the details covered above.