Feature Scaling for Machine Learning

Most of the linear classifiers you have met so far separate classes with a single flat boundary: a line in two dimensions, a plane in three. That restriction is simple and fast, but it is also a hard ceiling. Some datasets simply do not have their two classes arranged on opposite sides of any straight cut. This article starts with exactly that failure, shows where it appears in code, and then builds the tools that a kernelized support vector machine uses to get past it.

Begin by generating a small two-class dataset and fitting a linear support vector classifier to it. In this experiment the labels are produced by taking the cluster label modulo 2, which merges four blob centers into two classes. The classes are therefore not arranged as two clean halves, and that is the whole point of the experiment.

Run the code yourself and notice two things. The training accuracy is well below perfect. The model also reports exactly one weight per input feature plus one intercept. Those numbers describe a single straight line. A linear classifier cannot bend, so when the classes interleave, no choice of weights and intercept avoids the misclassified points. The source reports the same qualitative outcome: a linear model can separate these points only with a line, and it does not do a very good job on this dataset.

Each row of the original matrix is a point of the form (feature0, feature1). After the lift, each row is a point of the form (feature0, feature1, feature1 squared). The number of points stays the same. Only the number of coordinates grows. The lift is itself a one-line change, which appends the squared second feature as a new column of the input array.

The shape printout tells the story: the number of points is unchanged, but the number of features has grown by one. A linear SVM fitted to the expanded array finds a plane. If you project that plane back into the original two-dimensional plot, the boundary is no longer a straight line. It curves, and in the source's example it becomes something closer to an ellipse. This is the sense in which adding nonlinear features makes a linear model more powerful.

Adding a squared input works well in a small example, but the approach has a practical warning. Often you do not know which features to add, and the candidate list grows quickly. Every pairwise interaction in a 100-feature space, for instance, adds a large number of columns, which makes the computation expensive. You need a way to get the benefit of a high-dimensional representation without building it explicitly.

The kernel trick is the mathematical device that makes this possible. The learner needs only the scalar products between pairs of points in the expanded space, not the expanded coordinates themselves. A kernel is a function that takes two points in the original space and returns the value those scalar products would have had after expansion. The expansion is never written down.

The classifier only ever compares points through their pairwise scalar products. If a kernel can produce those products directly, the high-dimensional coordinates are never needed. That is why the source describes the trick as the reason kernelized SVMs are practical: it allows efficient computation in spaces that would otherwise be too large to handle.

Two kernels are commonly used with support vector machines. The polynomial kernel corresponds to all possible polynomials of the original features up to a chosen degree. The source's example of such a term is feature1 squared times feature2 to the fifth power. The radial basis function kernel, also called the Gaussian kernel, is harder to describe in finite terms because it corresponds to an infinite-dimensional feature space. One informal reading is that it considers polynomials of all degrees but gives less and less weight to the higher-degree features.

During training, the SVM estimates how important each training point is to the boundary between the two classes. In practice only a subset of those points matter. They are the points lying on the border between the classes, and they are called support vectors. The name of the method comes from them. The support vectors are the border points that define the boundary, and the remaining points relate to the boundary only through their absence from that definition.

Prediction works through distances. For a new point, the model measures its distance to each support vector and combines those distances with the learned importances. The source says these importances are stored in the dual_coef_ attribute of SVC. The Gaussian kernel supplies the distance measure.

Here $\lVert x_1 - x_2 \rVert$ is the Euclidean distance between two data points, and $\gamma$ (gamma) controls how quickly similarity fades with distance. The number of rows in support_vectors_ is the number of training points the model actually kept. Many fitted models will have far fewer support vectors than training points, which is the concrete form of the statement that only the border points matter.

Two parameters shape an RBF kernel SVM. Gamma sets the width of the Gaussian. A small gamma gives a large radius, so many points count as close to each other, and the boundary is smooth. A large gamma shrinks that radius, so the boundary bends around individual points and becomes more complex.

The parameter C is a regularization term, similar in spirit to the one in linear models. A small C restricts how much any single point can influence the model. Raising C lets misclassified points pull the boundary toward themselves, so the boundary bends to classify them.

The source applies an RBF kernel SVM to the Breast Cancer dataset with default settings. Here the failure is less visible than in the synthetic example, which is why it is worth tracing carefully. Fit the model on the raw training features and compare training and test accuracy.

Locate the point of divergence. The training score is perfect, but the test score is only 0.63. The model has memorized the training points and does not generalize. The first suspect is model complexity, but the more useful suspect is input scale. The source notes that the features of this dataset differ by completely different orders of magnitude. SVMs are very sensitive to the scaling of the data, because they generally need all features to vary on a similar scale.