Linear Models

Imagine you have been handed a very large table of records. Each row describes one item, and each column records one measurable property of that item. Your job is to sort every item into one of several categories. Before you write any code, you have to choose a learning method to try first, and two concerns usually compete. You may want a model that trains quickly even when the table is enormous, or you may want a model that squeezes out the last bit of predictive accuracy.

Naive Bayes sits on the speed side of that choice. This article explains what you gain by picking it and what you give up. It then turns to linear models, the family that Naive Bayes is often compared against, and builds up the prediction formula they all share, the geometry that formula produces, and the belief that linear models must be too simple to be useful.

The first thing to settle is the task type. If your goal is to answer a question such as which group an item belongs to, Naive Bayes is a candidate. Naive Bayes is a classification algorithm. It is not intended for numeric prediction, so if your goal is to estimate a number such as a price or a temperature, it is the wrong family of tool. Keeping this boundary in mind prevents the most basic selection error.

The main reason to choose Naive Bayes is speed. The source describes it as even faster than linear models. On a small dataset, the difference in running time may barely matter. On a huge dataset, the difference can decide whether an experiment is practical to repeat several times in one working day. A fast model lets you try ideas, inspect results, and adjust your data quickly.

The source reports this speed advantage as a comparative fact about the algorithm. It does not walk through the arithmetic that causes it in this material, so treat it as a result to remember: when you compare the two families on running time, Naive Bayes is the faster one.

The same speed advantage is why the algorithm is described as good for very large datasets. A method that is quick on a modest sample may become slow once the number of rows grows, while Naive Bayes is noted specifically for holding up well as the data scales.

Row count is only one dimension of size. Width is the other. A dataset is high-dimensional when each item is described by a very large number of features, meaning many columns rather than a few. The source states that Naive Bayes is good for high-dimensional data. In practical terms, a large number of input properties does not make it an unsuitable choice. It is one of the reasons the algorithm is worth knowing when your inputs are wide as well as long.

Speed comes with a cost. The source notes that Naive Bayes is often less accurate than linear models. The word often matters. It does not say the algorithm is always weaker, only that, as a general pattern, you should expect linear models to score higher on accuracy. This is the central trade-off: you accept some predictive quality in exchange for a faster and more scalable model.

For regression, the general prediction formula for a linear model is written as shown below. Here x[0] through x[p] are the features of a single data point, where the number of features is p. The values w and b are parameters learned from data, and the symbol ŷ, read as y-hat, is the prediction the model makes.

Read the formula from left to right as a recipe. Each feature is multiplied by its own weight, the products are added together, and the constant b is added at the end. Because the prediction is built from these multiplications and additions, it is called a weighted sum of the input features. The weights are the entries of w, and they can be negative. A negative weight means that a larger value of that feature pushes the prediction down rather than up.

The middle features follow the same repeating pattern as the first and last. Each one contributes its own weighted term, so adding a feature adds one more term to the sum and one more weight to learn.

When the dataset has only a single feature, the formula simplifies to the equation of a line, $$\hat{y} = w[0]\,x[0] + b$$ in the same notation, where w[0] is the slope and b is the intercept.

The two kinds of learned parameter have distinct geometric meanings. Once you know what each one controls, the learned values stop looking like arbitrary numbers and start describing a shape.

Take the one-feature line with slope 1 and offset 0. Making the slope larger steepens the line. Making it smaller flattens it. Making it negative tips the line downward, so the prediction falls as the feature grows. Changing b does not change the tilt at all. It slides the entire line up or down, which moves where it crosses the y-axis.

The before-and-after comparison shows two independent effects. The slope change alters how the line tilts, and the offset change alters where it sits. Because the two effects are separate, a model can shift its overall level without changing how sensitive it is to each feature.

The shape of a linear regression prediction depends on how many features the model uses. With one feature the prediction is a line. With two features it is a plane. With more features it is a hyperplane, which is the higher-dimensional version of a line or plane. The source uses hyperplane to describe the general case.

The formula does not change across these cases. Each added feature adds one term and one weight. What changes is the geometric object that the formula traces out in the space of the features.

A common first reaction to linear models is that they must be too simple. If you fit a straight line to a wiggly pattern, the line seems to ignore almost everything interesting in the data. The source agrees that on one-dimensional wave data, a straight line looks very restrictive, and that in a sense the fine details of the data are lost. It also calls the assumption that the target is a linear combination of the features a strong and somewhat unrealistic assumption. The opening belief is therefore partly correct.

The source then points out that judging linear models from one-dimensional data gives a skewed perspective. With many features, linear models can be very powerful. If there are more features than training data points, any target can be perfectly modeled as a linear function on the training set. The one-dimensional picture is misleading because a hyperplane in many dimensions has far more room to pass through the training targets than a single line does.