Polynomial Fit: Overfitting and Ridge
Interactive lab
Try it: Polynomial Fit: Overfitting and Ridge
How raising the polynomial degree drives training error down while validation error eventually rises (overfitting), and how a ridge penalty λ‖w‖² tames a high-degree fit.
How it works
- Split the noisy points into a training set and a validation set.
- Build the design matrix X with columns 1, t, t², …, t^d (t = 2x − 1, a rescaled x).
- Solve (XᵀX + λI)w = Xᵀy on the training points by Gaussian elimination with partial pivoting.
- Measure the mean squared error on the training points and, separately, on the validation points.
- Repeat for every degree 0–9: the degree with the lowest validation error is the one validation selects.
Default run (18 steps): 14 noisy points: 10 for training, 4 held out for validation. Fit a degree-3 polynomial with λ = 0. … Degree 3: training MSE 0.0282, validation MSE 0.0308. The lowest validation error is at degree 4. Training error keeps falling with degree; validation error is what reveals overfitting.
Simplified: One input, 14 fixed noisy samples of sin(2πx) and a single train/validation split (no cross-validation). The penalty λ‖w‖² here includes the constant term and is added to the sum (not the mean) of squared errors.
Educational simulation
Loading the simulation…