You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
After this lesson you will be able to: generate your own data, fit polynomials of degree 1, 3, and 9, and then see bias and variance in numbers: underfitting is wrong on both train and test, while overfitting has a training error near 0 but a test error that shoots up.
import numpy as np
rng = np.random.default_rng(2) # fixed seed → everyone gets the same results
def f(x): # true pattern: one period of a sine wave
return np.sin(2 * np.pi * x)
x_train = np.linspace(0, 1, 10)
y_train = f(x_train) + rng.normal(0, 0.3, 10) # 10 training samples, noise σ = 0.3
x_test = rng.uniform(0, 1, 200)
y_test = f(x_test) + rng.normal(0, 0.3, 200) # 200 previously unseen samples
def mse(y, y_hat):
return np.mean((y - y_hat) ** 2)
for degree in (1, 3, 9):
coef = np.polyfit(x_train, y_train, degree)
tr = mse(y_train, np.polyval(coef, x_train))
te = mse(y_test, np.polyval(coef, x_test))
print(f"degree {degree}: train = {tr:.3f} test = {te:.3f}")
# degree 1: train = 0.220 test = 0.353
# degree 3: train = 0.086 test = 0.146
# degree 9: train = 0.000 test = 0.914
With this data:
| Degree | Train | Test | Diagnosis |
|---|---|---|---|
| 1 | 0.220 | 0.353 | Underfitting (high bias): a straight line cannot bend along the sine wave, so it is wrong on both sets |
| 3 | 0.086 | 0.146 | Reasonable: the test error is close to the noise level σ² = 0.09, which no model can remove |
| 9 | 0.000 | 0.914 | Overfitting (high variance): 10 coefficients for exactly 10 points, so the curve passes through the noise as well |
train_err, test_err = [], []
for degree in range(1, 10):
coef = np.polyfit(x_train, y_train, degree)
train_err.append(round(float(mse(y_train, np.polyval(coef, x_train))), 3))
test_err.append(round(float(mse(y_test, np.polyval(coef, x_test))), 3))
print(train_err) # [0.22, 0.217, 0.086, 0.081, 0.077, 0.06, 0.06, 0.009, 0.0]
print(test_err) # [0.353, 0.359, 0.146, 0.154, 0.143, 0.158, 0.158, 0.501, 0.914]
The training error never increases as the degree increases (a higher-degree model contains the lower-degree models, so it fits train at least as well). The test error decreases and then rises again. In this run, the lowest test error is at degree 5, but it is almost the same as degree 3, so we choose degree 3 (simpler). This is specific data; running with a different seed will differ slightly.
for n in (10, 30, 100):
x = np.linspace(0, 1, n)
y = f(x) + rng.normal(0, 0.3, n)
coef = np.polyfit(x, y, 9)
print(n, round(mse(y, np.polyval(coef, x)), 3), round(mse(y_test, np.polyval(coef, x_test)), 3))
# 10 0.0 0.872
# 30 0.087 0.146
# 100 0.061 0.109
The degree-9 model does not change, but the more training samples there are, the lower the test error becomes (the first row differs from 0.914 because this is a fresh draw of noise). The model can no longer "memorize the noise" so easily.
Flashcards · 1.12 Lab: Bias-Variance Through Experiments
10 wordsNumPy function that fits a polynomial of a given degree and returns its coefficients.
Evaluates a polynomial with given coefficients at chosen x values to get predictions.
Controls model complexity: higher degree means a more flexible curve with more coefficients.
A model too simple for the pattern; error is high on both training and test data (high bias).
A model so flexible it fits the noise; training error near 0 but test error high (high variance).
The irreducible error from random noise; no model's test MSE can go meaningfully below it.
The difference between training and test error; a large gap is the signal of high variance.
With more training samples, a flexible model can no longer memorise the noise so easily.
Creates a random generator with a fixed seed so everyone gets the same synthetic data.
When two models have almost the same test error, prefer the less complex one.