Polynomial Features and Regression

A typical machine learning workflow is a chain of operations. Raw numbers are transformed, then transformed again, and only at the end is a model fitted and used to make predictions. Each link in that chain can be handled as a separate object, but then the code that runs the chain gets scattered across many lines. That spread makes it easy to apply the steps in the wrong order or to forget one of them.

A scikit-learn pipeline solves this by wrapping the whole chain in a single object. Instead of managing a scaler, a polynomial expander, and a regressor as three separate things, you treat them as one estimator. You give that one estimator the training data, and it passes the data through each step in turn. This article follows one concrete case from the source: earlier work computed polynomial features from the Boston dataset and passed them to a ridge regressor, and the same work can be modeled as one pipeline.

The case in the source has three steps: scaling the data, computing polynomial features, and applying ridge regression. Each step is either a transformer or an estimator. The scaler and the polynomial expander are transformers, because they change the input values. The ridge regressor is an estimator, because it is fitted to the target and makes predictions. The pipeline keeps all three in a fixed order.

The source builds this pipeline with the make_pipeline() function. You pass the steps as arguments, in the order they should run. The pipeline does not reorder them. The diagram below shows the chain for this case, read from left to right.

The source fragment does not show its imports, so the example below includes them. It also uses a small generated dataset in place of the Boston data, so the code runs on its own. The structure of the pipeline is the same either way. Read the code and notice how many objects the single pipeline variable is holding.

The pipeline holds three steps even though you wrote only one assignment. The stored steps can be inspected through pipe.steps or pipe.named_steps. The names are assigned from the class names, so each step can be found by the name the pipeline gave it.

When fit is called on the pipeline, the training data is handled one step at a time. The scaler learns what it needs from the data and transforms it. The polynomial step then expands the transformed values into polynomial terms. The ridge regressor is fitted on those expanded values together with the target. You do not write any of this sequencing yourself. The pipeline's fit method carries it out in the stored order.

The source describes a second, more important benefit of wrapping the steps together. Because the preprocessing and the model live in one object, the parameters of the preprocessing can be adjusted using the outcome of a supervised task, such as a regression or a classification. The number of polynomial terms is one such preprocessing parameter. Its value is judged by how well the final model predicts the target.

Without a pipeline, a learner might choose the number of polynomial terms by hand, fit the model, and then repeat the process. Nothing in the code ties those choices together, so the link between the preprocessing decision and the target is easy to lose. With a pipeline, the preprocessing steps are part of what gets evaluated against the target. The comparison below sets the two ways of working side by side.

The main difference is not in the arithmetic. Each step computes the same thing whether it is called inside a pipeline or manually. The difference lies in who sequences the work and who keeps the choices connected. When you apply the steps one at a time, you decide the order, you call each fit and transform yourself, and you carry the results forward. When you use the pipeline, the stored order is enforced for you, and fit and predict on the whole object run through every step.