Logistic Regression and Linear Support Vector Classifiers

Naive Bayes classifiers belong to the same broad family as the linear models you have already studied. Their decision rules take a similar shape, and they usually reach a slightly lower generalization accuracy than classifiers such as LogisticRegression and LinearSVC. What they give up in accuracy they recover in speed: Naive Bayes models are noticeably faster to train than those linear methods, and they are very fast to predict as well.

Before any theory, start with a concrete dataset. Imagine four data points, each described by four binary features, where every feature is either 0 or 1. Each data point also carries a class label, and there are two classes. The raw material a Naive Bayes model works with is simply a tally of how often each feature equals 1 within each class. Keep that in mind as you read the table below, because every later idea in this article is built from those tallies.

A Naive Bayes model does not look at whole rows at once during training. Instead it takes each feature separately and gathers simple statistics for every class. For each class, it asks: across all rows that belong to this class, how many times does this feature equal 1? In the dataset above, class 0 contains the first two rows and class 1 contains the last two.

Read the diagram as a lookup table. The value under Feature 0 for class 0 is 2 because both of its rows have a 1 in that column. The value under Feature 0 for class 1 is 0 because neither class 1 row has a 1 there. These per-class tallies are the entire set of statistics the model needs from the training data.

The speed comes from how the model learns. A linear model such as LogisticRegression optimizes all of its feature weights together. A Naive Bayes model instead looks at each feature on its own and gathers simple statistics for every class. Because the work for each feature is independent and cheap, the whole training process stays short. The word naive in the name refers to this simplifying choice to treat features separately.

Training a Naive Bayes model is a counting exercise. The model reads the rows once, tallies feature values per class, and stores those tallies. There is no iterative search over weights that repeatedly revisits the data. This single pass is why training is fast, and it is also why Naive Bayes stays fast on very large datasets where training even a linear model might take too long.

Fit therefore needs only one pass over the data, and it touches each feature in a simple way. Predict then reuses the stored counts rather than recomputing anything from the training rows. Both phases are cheap, which explains why the source describes Naive Bayes models as very fast to predict as well as to train.

scikit-learn implements three Naive Bayes classifiers. They differ mainly in the type of data they assume. GaussianNB is meant for continuous data of any kind. BernoulliNB assumes each feature is binary, meaning it is either present or absent. MultinomialNB assumes count data, where each feature is an integer count of something, such as how often a particular word appears in a sentence.

The assumed data type changes the per-class distribution the model learns. GaussianNB summarizes each feature within each class as a continuous spread. BernoulliNB records, for each class, how often each binary feature is present. MultinomialNB records, for each class, how the integer counts of each feature are distributed. In every case the learned statistics are still simple per-class summaries, which is what keeps the training fast.

MultinomialNB and BernoulliNB each have one main tuning parameter, alpha, which controls model complexity. The mechanism is smoothing. The algorithm adds alpha virtual data points that have positive values for all features, which softens the statistics it collected from the real data. A larger alpha means more smoothing and a less complex model.

Smoothing matters because raw counts can be zero. In the example table, class 1 never has a 1 in feature 0, and class 0 never has a 1 in feature 3. A raw count of zero produces a raw probability of zero, which would let a single missing feature veto a class entirely. Adding alpha to every count keeps every estimate above zero.

Naive Bayes models share many strengths and weaknesses with linear models. They train and predict very quickly, the training procedure is easy to follow, and they handle high-dimensional sparse data well. They are also relatively robust to their parameters. Because of this, they serve as strong baseline models.

Among the variants, MultinomialNB usually performs better than BernoulliNB, particularly on datasets with a relatively large number of nonzero features, such as large documents. GaussianNB is mostly used for very high-dimensional data, while the other two variants are widely used for sparse count data such as text.