Model Evaluation and Learning Curves

This article starts with a prediction. Ridge Regression is a linear model for regression, so it makes predictions with the same formula that ordinary least squares uses. What separates it from a plain LinearRegression model is how its coefficients are chosen, and that difference shows up first in the scores the two models produce on the same data. Before you look at any output, decide what you expect the training score and the test score to look like for Ridge compared with LinearRegression.

Ordinary Linear Regression chooses its coefficients with one goal: predict the training data as well as possible. Ridge Regression keeps that goal and adds a second one. The second goal is a constraint on the coefficients themselves: keep their magnitudes as small as possible, so that every entry of the coefficient vector is close to zero. In intuitive terms, each feature should have as little effect on the outcome as it can, which corresponds to a small slope, while the model still predicts well.

The constraint is not something Ridge discovers from the data. It is a deliberate design choice built into the model, which is why it can be described as restricting the model explicitly. In scikit-learn the model is available as linear_model.Ridge, and the code in the next section uses that class. Because both models use the same prediction formula, the only thing that changes when you switch from LinearRegression to Ridge is which coefficient values are selected during training.

When an ordinary linear model is trained on the housing data used in the source, it fits the training set very closely. That close fit is a warning sign. The model has been overfitting, memorizing details of the training data that do not carry over to new data. A high training score paired with a weaker test score is the signature of this problem.

Ridge changes that trade. Its training score drops, because the model is no longer allowed to bend freely to every training point. Its test score rises, because the simpler model generalizes better. The source states the logic directly: a less complex model gives worse performance on the training set but better generalization, and if your only interest is generalization, you should prefer the Ridge model over LinearRegression.

The amount of weight Ridge gives to simplicity compared with training set performance is set by the alpha parameter. You specify it yourself. The default, alpha=1.0, is used in the first example, but there is no reason it should give the best trade-off. The optimum alpha depends on the particular dataset, so it has to be chosen for the data at hand.

The direction of the effect is consistent. Increasing alpha pushes the coefficients toward zero. That decreases training set performance, but it may help generalization. Decreasing alpha lets the coefficients be less restricted, and for very small values of alpha the model ends up resembling LinearRegression. Think of alpha as a dial with simplicity at one end and training fit at the other.

The coefficient plot in the source confirms this picture. For alpha=10 the coefficients sit mostly between about -3 and 3. The coefficients for alpha=1 are somewhat larger, those for alpha=0.1 are larger still, and many coefficients from unregularized linear regression fall outside the chart altogether. A higher alpha means a more restricted model with smaller coefficient magnitudes.

Notice that the code above does not claim any particular output. Run it on your own split and read the three scores side by side. You are looking for the pattern the source describes: the linear model's training score is highest, Ridge's training score is lower, and the test scores decide which model generalizes better.

Another way to see the influence of regularization is to hold alpha fixed and vary how much training data is available. Plots that show model performance as a function of dataset size are called learning curves. In the source, the Boston Housing dataset was subsampled, and LinearRegression and Ridge with alpha=1 were evaluated on subsets of increasing size.

Read the curves in three parts. First, the training score is higher than the test score at every dataset size for both models, which is the normal pattern. Second, Ridge's training score stays below LinearRegression's at every size, because Ridge is regularized. Third, Ridge's test score is better, especially for small subsets. For fewer than about 400 data points, LinearRegression cannot learn anything useful, while Ridge can. As more data arrives, both models improve, and LinearRegression catches up with Ridge by the end.

The LinearRegression curve also shows its training score falling as data is added. With more data it becomes harder for a model to overfit, or memorize, the examples it sees. So regularization matters most when training data is limited: the gap between a flexible model and a restricted one is largest when there are few examples to constrain the fit.

Lasso is an alternative to Ridge for regularizing linear regression. Like Ridge, it restricts the coefficients to be close to zero, but it does so in a slightly different way, called L1 regularization. The important consequence is that some coefficients can become exactly zero. Those features are ignored by the model, which can be seen as a form of automatic feature selection.

The source's recommendation follows from this contrast. In practice Ridge is usually the first choice between the two. If you have a large number of features and expect only a few to matter, Lasso may be the better choice, and it also suits a situation where you want an easy-to-interpret model. scikit-learn additionally provides ElasticNet, which combines the Lasso and Ridge penalties; that combination often works best, at the cost of tuning two parameters.