Linear Models for Regression

Many introductory machine learning methods answer the question of which category a new data point belongs to. Regression asks a different question: what number should we predict? A house price, a temperature, or a score is a continuous target value, and a regression model must produce a number for every new input.

This article covers two families of regression models. The first is k-nearest neighbors regression, which keeps the familiar idea of looking at the closest training points but uses their numeric targets to produce a prediction. The second is the linear regression model, which predicts a weighted sum of the input features plus an offset. For both, we will trace exactly what the model computes, so that its predictions stop looking like magic.

The simplest case is a single neighbor. To predict the target for a new input, the algorithm finds the nearest training point and returns that point's target value. Nothing is averaged. The prediction is copied directly from the closest example.

When more than one neighbor is used, the prediction becomes the average, or mean, of the target values of the relevant neighbors. The number of neighbors is set by the parameter n_neighbors, which is often written as k. The steps are the same for every value of k: find the k nearest training points, read their target values, and average them. The diagram traces this path for one query point.

In scikit-learn, KNeighborsRegressor is used the same way as KNeighborsClassifier. You create it with n_neighbors, call fit on training data, call predict on new inputs, and call score to get the R2 score. The program below follows the four jobs a complete regression run needs: load data, split it, fit a regressor with three neighbors, then predict and score on the test set. The dataset here is a small synthetic sine-shaped set generated for this lesson.

Read the program as one complete unit. The line that fits the model uses only the training split. Only afterward do we call predict and score on the test split. If you accidentally fit on the full dataset before scoring, the score would reflect inputs the model had already seen, and it would no longer be a fair measure of how the model handles new data.

The number of neighbors controls how closely the model follows the training data. The source compares predictions for 1, 3, and 9 neighbors on a one-dimensional dataset, evaluating the model along a dense line of inputs across the feature range. The comparison exposes the central trade-off in choosing k.

With a single neighbor, the predicted values pass through all of the data points, which gives a very unsteady prediction. Considering more neighbors produces smoother predictions, but those smoother predictions fit the training data less well. A larger k smooths the curve because each prediction averages more targets, so one unusually high or low training target has less influence on any single output.

The source identifies two main parameters for k-nearest neighbors: the number of neighbors and the way distance between data points is measured. By default, KNeighborsRegressor uses Euclidean distance, which works well in many settings. The number of neighbors is the parameter you should adjust.

To read a score, compare it against those two reference points. A score near 1 means the predictions closely match the targets. A score near 0 means the model is no better than always predicting the training mean. The source reports a test score of 0.83 for its model, which it describes as indicating a relatively good model fit.

A linear model for regression predicts a weighted sum of the input features plus an offset. For a single data point with features x_0 through x_p, where p is the number of features, the general prediction formula is written below. The values w and b are parameters learned from data, and the symbol ŷ, read as y-hat, is the prediction the model makes.

Read the formula from left to right as a recipe. Each feature is multiplied by its weight, the products are added, and the constant b is added at the end. The weights are the entries of w, and they can be negative. A negative weight means that a larger value of that feature pushes the prediction down rather than up. Adding a feature adds one more term to the sum and one more weight to learn.

The two kinds of learned parameter have distinct geometric meanings. With a single feature, the parameters describe a line, and the learned values are interpreted directly from it. The parameter meanings below follow from that picture.

The shape of a linear regression prediction depends on how many features the model uses. The source characterizes linear models by the geometric object their prediction forms. The table summarizes the cases.

A hyperplane is the higher-dimensional version of a line or plane, and the source uses it to describe the general case. You cannot draw a hyperplane for many features, but the same formula and the same role for each weight and the offset apply in every dimension.

A common first reaction to linear models is that they must be too simple. A straight line fitted to a wiggly pattern seems to ignore almost everything interesting in the data, and many learners conclude that a model which outputs only a line or a flat surface cannot capture real structure. The source agrees that, on one-dimensional data, a straight line looks very restrictive, and that in a sense the fine details of the data are lost. It also calls the assumption that the target is a linear combination of the features a strong and somewhat unrealistic assumption.

The source also notes that there are many different linear models for regression. They all share the same prediction formula. What differs between them is how the parameters w and b are learned from training data and how model complexity can be controlled. The source says it will go on to cover the most popular of these models.