Linear Models for Classification and Regression

Imagine you have just learned the k-Nearest Neighbors method and want to know whether it is worth using on a real problem. The answer depends on a trade-off. The method is one of the easiest supervised learning ideas to explain, and it often produces reasonable results without much tuning. At the same time, the way it makes predictions can become expensive as your data grows, and it can struggle when each data point carries many features. This article asks a practical question: when does the simplicity of k-NN pay off, and when does it become a liability?

The second half of the article moves to linear models. They avoid the two drawbacks that limit k-NN, and they are the family behind LogisticRegression and LinearSVC. You will follow a single mechanism step by step: a weighted sum of features, an intercept, and a threshold at zero. Keep the k-NN trade-off in mind as you go, because it explains why linear models are often the next thing to try.

In principle, a k-NN classifier has two important parameters. The first is the number of neighbors, which is the value k in the name of the method. The second is the way you measure distance between data points. These two choices work together: the number of neighbors decides how many nearby training examples take part in a vote or average, and the distance measure decides what counts as nearby in the first place.

For the number of neighbors, practical experience suggests that a small value such as three or five often works well. The source is clear, however, that you should still adjust this parameter for your own data rather than accept a default guess. For the distance measure, the default is Euclidean distance, which works well in many settings. The source notes that choosing the right distance measure is somewhat beyond its scope, so for now treat Euclidean distance as the standard starting point.

Because of these properties, k-NN is a good baseline method. A baseline is a simple reference point: if a more advanced technique cannot beat it, the extra complexity is probably not worth the cost. Trying k-NN first gives you a clear number to compare against.

The main cost of k-NN appears at prediction time. Building the model is usually quick, but when the training set is very large, prediction can become slow. The size can be measured in two ways: the number of features, or the number of samples. A training set that is large in either dimension makes each prediction more expensive, because the new point has to be compared against many stored examples.

k-NN also does not perform well on datasets with many features, and it does particularly badly on sparse datasets, where most features are 0 most of the time. Each prediction compares the new point against every stored row, so with hundreds of features each comparison involves far more values, and the distances may stop being informative. This is why the source concludes that k-NN is seldom the final choice even though the code looks short and simple.

k-NN decides everything by comparing distances, so the scale and quality of the input values feed directly into the result. The source therefore states that when using k-NN, it is important to preprocess your data. Preprocessing means preparing the raw values before they reach the model, so that the distances the algorithm computes reflect the patterns you care about. Features measured on very different scales can dominate the distance calculation, which is why preprocessing matters before you trust a k-NN result.

A linear model for binary classification starts with the same kind of calculation as a linear regression model. It forms a weighted sum of the input features and adds an intercept. Here $w$ holds the coefficients, $x$ holds the features, and $b$ is the intercept.

The difference lies in the final step. A regression model returns this weighted sum as its answer. A linear classifier thresholds the value at zero. If the value is smaller than zero, the model predicts class -1. If the value is larger than zero, it predicts class +1. This prediction rule is common to all linear models for classification. The models differ in how they find the coefficients $w$ and the intercept $b$, not in how they turn the score into a class.

The rule above creates a boundary between the two classes. For regression, the output is a linear function of the features, which is a line, a plane, or a hyperplane in higher dimensions. For classification, the decision boundary is the set of points where the score sits exactly at the threshold. A binary linear classifier therefore separates two classes using a line, a plane, or a hyperplane. Any new point on one side of that boundary gets one class, and any point on the other side gets the other class.

The source illustrates this with the forge dataset. Both LinearSVC and LogisticRegression are fit with their default parameters, and each produces a straight line. The area on one side of the line is classified as class 1 and the area on the other side as class 0. The two lines are similar, and both misclassify two training points. That agreement is the point of the example: two different algorithms can learn very similar boundaries from the same data.

A common first reaction to the name LogisticRegression is to assume it predicts a number, the way LinearRegression does. That assumption is wrong. LogisticRegression is a classification algorithm, not a regression algorithm, even though its name suggests otherwise. Linear models are used heavily for classification, meaning they predict which category an input belongs to, and LogisticRegression is one of the two most common linear classification algorithms.

The two most common linear classification algorithms are logistic regression, available as LogisticRegression in the linear_model module, and linear support vector machines, available as LinearSVC in the svm module. The name SVC stands for support vector classifier. Both are linear classifiers, and both apply L2 regularization by default. Once you see how a linear model turns a score into a class label, the name stops being confusing.

The many algorithms for learning linear models differ in two ways. The first is how they measure how well a particular combination of coefficients and intercept fits the training data, which is the loss function. The source notes that this choice matters little for many applications. The second is whether they use regularization, and if so, what kind. Regularization is the choice that matters most for controlling how a model generalizes.

For both LogisticRegression and LinearSVC, the parameter that sets the strength of regularization is called C. Higher values of C correspond to less regularization. A high C asks the model to fit the training set as well as possible. A low C asks the model to keep the coefficient vector close to zero. So C is a single dial that trades off fitting the training data against keeping the model simple. A low C pushes coefficients toward zero, while a higher C allows a closer fit to the training set.

The source also describes a second way C acts. A low C makes the algorithm adjust to the majority of the data points, while a higher C gives more weight to classifying each individual point correctly. The source illustrates this with LinearSVC on the forge dataset across three values of C. With a very small C, the model chooses a relatively horizontal line and misclassifies two points. With a slightly higher C, the line tilts to respond to the two misclassified samples. With a very high C, the line tilts sharply and classifies all class 0 points correctly, but still misclassifies one class 1 point, because no straight line can separate this dataset perfectly.

The source warns that this most aggressive model is likely overfitting, since it tries hard to classify every point and may miss the overall layout of the classes. A very high C can overfit, so judge the value of C by comparing training and test accuracy rather than training accuracy alone. The source reports, for an L1 penalty on the same kind of task, training and test accuracies of 0.91 and 0.92 for C equal to 0.001, 0.96 and 0.96 for C equal to 1, and 0.99 and 0.98 for C equal to 100. The training accuracy keeps rising with C, but the test accuracy is the signal that tells you when the extra fit stops helping.