Enhancing Linear Models with Interactions and Polynomials

Suppose a straight line is a poor description of your data. The points curve, rise, and flatten, and no single straight line passes near them. A plain linear regression model can only draw a straight line through the input, so its predictions will miss the shape of the data no matter how carefully it is fitted. Polynomial regression is one way to keep the linear model machinery while letting the prediction bend.

The idea is easy to state. Polynomial regression is linear regression applied to polynomial features. The fitting method does not change. What changes is the input the model receives. This article follows one one-dimensional example from raw input to curve, and it asks at each step whether the fitted curve still matches what you expected. Finding the step where that match breaks is the main skill this article builds.

A polynomial regression pipeline has two lanes that meet at the prediction. The training lane takes the raw input, passes it through a polynomial transformer to produce a feature matrix, and then fits the linear model to that matrix together with the target. The drawing lane takes a dense grid of input values, passes it through the same transformer, and then applies the fitted weights to produce the curve.

Read the diagram from left to right in each lane. The two lanes must use the same transformer and the same column layout. If they do not, the weights learned from the training columns will be applied to the wrong columns in the drawing lane, and the curve will be wrong even though the fit itself succeeded.

A polynomial feature is a transformed version of the original input. Instead of giving the linear model only the raw value of the input feature, the transformer builds new columns from that value, using higher powers of it up to a chosen degree. The source example uses polynomial features of degree ten, meaning the highest power in the new columns is ten.

The diagram shows one row of expanded columns for one input value. The first column holds the constant 1, the second holds the raw value, the third holds its square, and the fourth holds its cube. The linear model then fits one weight to each column. The prediction for a row is the weighted sum of these columns, so the model is still linear in its weights even though the output traces a curve in the original input.

The linear regression model itself never sees a curve. It sees a table of numbers and fits weights to that table. The bending is produced entirely by the feature columns that were prepared before fitting. This separation is the central idea of the method, and most debugging later depends on keeping the two steps distinct: the transformer decides what columns exist, and the linear model decides how much each column counts.

A linear regression model and a polynomial regression model share the same fitting method. They differ in what the model receives as input and, as a result, in the shape of the prediction they can represent. The left path in a comparison skips feature preparation, so the prediction is limited to what a straight line can express. The right path inserts polynomial features before fitting. The fitting step is identical, so polynomial regression is not a different learning algorithm. It is the same linear model given a richer set of inputs.

The source compares polynomial regression with a kernel SVM learned on the original data, with no explicit transformation of the features. A kernel SVM uses a kernel to learn a prediction that is similarly complex to the polynomial regression result. The contrast is in where the higher-order behavior is computed. The polynomial version builds its expanded columns explicitly, before fitting. The kernel SVM reaches a comparable complexity without that explicit step.

The kernel SVM is a more complex model than the linear regression used here, and the source presents it as an alternative to compare against rather than a replacement for understanding polynomial features. The source's comparison varies the gamma parameter of an SVM with an RBF kernel, which is where the kernel SVM's behavior is tuned. Understanding the explicit polynomial pipeline first makes the kernel version easier to reason about, because you can ask the same question of both: where is the bending produced, and which setting controls how much of it appears?

On one-dimensional data, polynomial features give a very smooth fit. The curve does not zigzag from point to point the way a flexible model might, and that smoothness is part of the appeal of the method. The smoothness does not hold everywhere, however. High-degree polynomials can behave extremely at boundaries and in sparse regions, where there are few training points to constrain them.

This is why a prediction that looks correct in the middle of the plot can be wrong at its ends. When you inspect a polynomial regression result, check the boundaries and sparse regions first, not only the area where the points are densest. A curve that matches the data well in the center can still be unreliable where you have no evidence.

Most polynomial regression bugs show up as a prediction that disagrees with what you expected. The useful habit is to locate the step where the disagreement begins, rather than rewriting the whole pipeline. The pipeline has three checkpoints: the features the transformer produced, the model fitted on them, and the features passed in at prediction time.