Polynomial Features in Model Evaluation

This article begins at the point where a grid search has already finished running. The search compared several combinations of settings for a model that includes polynomial features and a ridge regression step, and it recorded which combination performed best under cross-validation. Your job is not to run the search again. Your job is to read the results carefully, check them against what you expect, and locate the exact line of output that answers each question.

The search stored its findings on an object named grid. Two attributes of that object do most of the work in this section. The first is grid.best_params_, which holds the winning combination of settings. The second is grid.score(X_test, y_test), which measures how well the selected model does on test data. These two answer different questions. The first tells you which settings won the search. The second tells you how the winning model performs on data it did not see during the search. Keeping them apart is the first habit to build.

The best parameters are reported as a dictionary with two keys: polynomialfeatures__degree and ridge__alpha. The first key sets the degree of the polynomial features, and the search chose the value 2. The second key sets the regularization strength for the ridge step, and the search chose 10. Each key names a component of the model and then the setting being tuned. This naming scheme is why the search can tune settings that belong to different parts of the same model at once.

The test-set score measures the chosen model on held-out data and is separate from the cross-validation results that selected the settings. The cross-validation results are the evidence used to pick degree 2 and alpha 10. The test score is a separate measurement taken after that choice has been made, on data that played no part in choosing the settings. This is why the two lines of output are not the same kind of number, even though both describe model quality.

Read the diagram from left to right. The test data only enters at the scoring step. Everything to the left of that step belongs to choosing settings. If you were to score the model on the same data that guided the choice, the number would look more favorable than it should, because the settings were tuned toward that data. Reporting the test score separately keeps the final measurement honest.

Polynomial features take an input and add new columns built from powers of that input. As the degree increases, the model receives more columns and can bend its fitted curve in more ways. The source's setting of polynomialfeatures__degree controls how many powers are created. Degree 2 keeps the original feature and adds its square, which is one reason the search can compare simple and flexible versions of the same input.

Ridge alpha works against this flexibility. The source sets ridge__alpha to 10 in the winning combination. Alpha controls the regularization strength of the ridge step. Stronger regularization discourages the model from relying heavily on any single large coefficient, which limits how wildly the fitted curve can bend. A high degree paired with a weak penalty can produce a curve that follows noise, and a high degree paired with a stronger penalty is held in check. The search tests these pairs together, which is why degree and alpha are read as a combination rather than as two separate choices.

The source states that degree three performs much worse than degree one or two. The winning degree in this search is 2. Degree 1 and degree 2 are the two strong candidates, and degree 3 is the clear loser. Notice what this comparison shows. More flexibility was available at degree 3, yet it produced worse evaluation results. Flexibility that the data cannot support does not improve the score.

Two claims are safe to make from the source. Degree 2 is the winner, and degree 3 is much worse than degrees 1 and 2. The source does not give a full table of scores for every degree, so do not invent exact numbers for the degrees it does not report. When you read your own search output, the same reasoning applies: find the best row, then check how far the other rows fall below it before concluding that a setting matters.

Sometimes an output disagrees with what you expected. The source's advice is to identify which stage produced that line before changing the code. Each stage in the path hands one result to the next, so a wrong-looking number can be traced backward until your expectation and the output first diverge.