Polynomial Features in Linear Models
The offsets come from the indicator columns and the slopes come from the product columns. Each bin ends up with its own offset and its own slope, even though the model itself is an ordinary linear regression.
Predicting the Shape of the Combined Matrix
A linear model that works on one x-axis feature can only draw one straight line. To let the line change across the range of x, the input is rebuilt as a wider feature matrix. Each row is still one data point, but the columns now record two kinds of information at once: which bin the point falls into, and the original x-axis value restricted to that bin. Before looking at how this matrix is built, it helps to predict its shape.
What do you think happens?
The data has 100 points. The matrix is built from two families of columns, and the combined matrix is printed with its shape. What shape do you expect?
Reveal answer
(100, 20)
The source reports the printed shape as (100, 20): 100 rows, one per data point, and 20 columns in the new representation. The first number is the number of data points and the second is the number of features, so the rows are never mixed up with the columns.
Two Families of Columns
The 20 features come in two families. The first family is the bin indicators, which record which bin each data point falls into. The second family is the product features. Each product feature takes the original x-axis value and keeps it only inside one bin; everywhere else it is zero. Both families are concatenated into one matrix, so the linear model sees a single set of columns and fits them together. Nothing in the model itself is special about bins: the structure comes entirely from how the columns are built.
Reading the layout row by row makes the meaning concrete. In a row where the point falls into bin 0, the bin 0 indicator column holds 1, the bin 1 indicator column holds 0, the bin 0 product column holds the x value, and the bin 1 product column holds 0. The indicator columns say where the point is; the product columns carry its x value into the bin it occupies.
Assembling the Feature Matrix
The construction has three stages: build the indicators, build the products, and stack them. The example below uses four points and two bins so that every intermediate array is small enough to print and check. It is an original illustration, not the source's data.
How Each Bin Gets Its Own Offset and Slope
When a linear regression is fit on the combined matrix, the result is a separate line for each bin. The lines do not have to share a slope. The model is still linear in its features, but the product features let the slope change from one bin to the next. The fit assigns one coefficient to each indicator and one coefficient to each product column.
- 1I_0 and I_1 are the bin indicator columns, each equal to 1 for points in its bin and 0 otherwise.
- 2The coefficients beta_0 and beta_1 multiply the indicators, so they act as the offset for each bin.
- 3The product columns x times I_0 and x times I_1 carry x only inside its bin, and the coefficients gamma_0 and gamma_1 multiply them, so they act as the slope for each bin.
- 4For a point in bin k, every term except the bin k indicator and bin k product is zero, so the prediction reduces to beta_k plus gamma_k times x.
The offsets come from the indicator columns and the slopes come from the product columns. Each bin ends up with its own offset and its own slope, even though the model itself is an ordinary linear regression.
Mistakes with Shapes and Stacking
X = np.hstack([bin0, bin1, x * bin0, x * bin1])
print(X.shape)Each array is one-dimensional with length 4, so np.hstack joins them end to end into a single long vector with shape (16,).
X = np.hstack([bin0[:, None], bin1[:, None], (x * bin0)[:, None], (x * bin1)[:, None]])
print(X.shape)Each array is a column with shape (4, 1), so np.hstack joins them side by side into a matrix with shape (4, 4).
Common mistake Forgetting to turn one-dimensional arrays into columns before stacking
What it looks likenp.hstack([bin0, bin1]) on two length-4 arraysWhy it failsOne-dimensional arrays are joined end to end, so the result has length 8 instead of having one column per bin across 4 rows.
Do this insteadAdd a column axis first, as in bin0[:, None], so each block has shape (4, 1) before stacking.
Common mistake Building product features from the full x values without the bin indicator
What it looks likex * 1 used as a product column
Why it failsA product feature is only the x value inside its own bin and zero elsewhere. Without the indicator, every row carries x in that column, so the column no longer restricts the slope to one bin.
Do this insteadMultiply x by the indicator for that bin, so the product is zero outside the bin.
Common mistake Checking only the final shape
Why it failsThe final shape can look right even when an earlier block has the wrong number of rows, which hides where the error started.
Locating Where the Matrix Goes Wrong
When the final shape is wrong, checking only the last line does not show the cause. The fix is to print the shape after each construction step and compare it with the shape you expect. The table below lists the checkpoints for the four-point example. The first row whose shape differs from the expected value is the step to inspect.
| Step | Expected shape | What to check |
|---|---|---|
| x values | (4,) | one value per data point |
| bin0 and bin1 indicators | (4,) | each is 0 or 1 for every row |
| bin0[:, None] and bin1[:, None] | (4, 1) | a column, so stacking is side by side |
| indicators from np.hstack | (4, 2) | one column per bin |
| products from np.hstack | (4, 2) | one column per bin, zero outside the bin |
| combined X from np.hstack | (4, 4) | same row count as the inputs |
Shape checkpoints for the four-point, two-bin example
Predicting the Width of a New Matrix
A feature matrix has an indicator block of shape (100, 10) and a product block of shape (100, 10). What shape does np.hstack return, and which columns hold the product features?
Hints
- Both blocks have the same number of rows, so the row count stays 100.
- The column counts add when blocks are joined side by side.
- The indicator block occupies the first 10 columns, so the product block starts after them.
Key Takeaways
- The interaction representation stacks bin indicators and product features into one matrix. In the source example this matrix has shape (100, 20): 100 data points and 20 columns.
- Bin indicators record which bin each data point falls into, and each product feature is the original x value inside its bin and zero everywhere else.
- Elementwise multiplication of x by a bin indicator produces the product feature, and
np.hstackjoins blocks side by side into one matrix. - A linear model fit on this representation gives each bin its own offset, from the indicator coefficients, and its own slope, from the product coefficients.
- Checking the shape after each construction step shows where a feature matrix first goes wrong.