Linear Models for Classification

Most of the linear classifiers you have met so far separate classes with a single flat boundary: a line in two dimensions, a plane in three. That restriction is simple and fast, but it is also a hard ceiling. Some datasets simply do not have their two classes arranged on opposite sides of any straight cut. This article starts with exactly that failure, shows where it appears in code, and then builds the tools a kernelized support vector machine uses to get past it.

Keep one question in mind as you read: when a model's answer looks wrong, where in the pipeline did the answer first diverge from what you expected? Several mistakes in this article are not errors in the algorithm itself but in the data representation or the scale of the inputs. Learning to locate the point of divergence is part of the skill.

Begin with a small two-class dataset and fit a linear support vector classifier to it. The labels are produced by taking the cluster label modulo 2, which merges four blob centers into two classes. The classes are therefore not arranged as two clean halves, and that is the point of the experiment.

Run the code and notice two things. The training accuracy is well below perfect, and the model reports exactly one weight per input feature plus one intercept. Those three numbers describe a single straight line. A linear classifier cannot bend, so when the classes interleave, no choice of weights and intercept avoids every misclassified point. The source reports the same qualitative outcome: a linear model can only separate these points with a line and does not do a very good job on this dataset.

The reveal is worth dwelling on. Linearity is a property relative to the coordinates you give the model. Change the coordinates and the same family of models behaves differently. Adding derived features such as a squared input can turn a non-separable problem into one that a plane can separate. The catch is that choosing those features by hand is difficult and expensive.

Read the diagram from top to bottom. On the left, each point is described by two coordinates and no line separates the classes. On the right, the same points are described by three coordinates, where the third is the square of the second. The number of points is unchanged, but the number of features has grown by one. A flat plane in this expanded space separates the classes. When that plane is projected back into the original two-dimensional plot, the boundary is no longer straight. In the source's example it curves into something closer to an ellipse. This is the sense in which nonlinear features make a linear model more powerful.

The kernel trick is the mathematical device that makes this possible. The learner needs only the scalar products between pairs of points in the expanded space, not the expanded coordinates themselves. A kernel is a function that takes two points in the original space and returns the value those scalar products would have had after expansion. The expansion is never written down.

The two columns of the comparison reach the same quantity by different routes. The explicit route builds each expanded vector and then takes scalar products. The kernel route skips the expanded vectors and computes the scalar product directly from the original points. The classifier only ever compares points through these pairwise scalar products, so when a kernel can produce them directly, the high-dimensional coordinates are never needed. The source describes this as the reason kernelized SVMs are practical: it allows efficient computation in spaces that would otherwise be too large to handle.

Two kernels are commonly used with support vector machines. The polynomial kernel corresponds to all possible polynomials of the original features up to a chosen degree. The source's example of such a term is feature1 squared times feature2 to the fifth power. The radial basis function kernel, also called the Gaussian kernel, corresponds to an infinite-dimensional feature space and is harder to describe in finite terms. One informal way to read it is that it considers polynomials of all degrees, while giving less and less weight to features at higher degrees.

Here $\lVert x_1 - x_2 \rVert$ is the Euclidean distance between the two data points, and $\gamma$ (gamma) controls how quickly similarity fades with that distance. The source notes that the detailed mathematics of kernel SVMs is not essential for using them. What matters in practice is how the model makes a decision, which the next section explains.

During training, the SVM estimates how important each training point is to the boundary between the two classes. In practice only a subset of those points matter. They are the points lying on the border between the classes, and they are called support vectors. The name of the method comes from them.

Prediction works through distances. For a new point, the model measures its distance to each support vector and combines those distances with the learned importances. The source says these importances are stored in the dual_coef_ attribute of SVC. The Gaussian kernel supplies the distance measure. The number of rows in support_vectors_ is the number of training points the model actually kept. Many fitted models have far fewer support vectors than training points, which is the concrete form of the statement that only border points matter.

Two parameters shape an RBF kernel SVM. Gamma sets the width of the Gaussian. A small gamma gives a large radius, so many points count as close to each other, and the boundary is smooth. A large gamma shrinks that radius, so the boundary bends around individual points and becomes more complex. The parameter C is a regularization term, similar in spirit to the one in linear models. A small C restricts how much any single point can influence the model. Raising C lets misclassified points pull the boundary toward themselves, so the boundary bends to classify them.