Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1: Data Preparation and Model Foundations

1.12 Lab: Bias-Variance Through Experiments

Print / Save as PDF

Lab: Bias-Variance Through Experiments

After this lesson you will be able to: generate your own data, fit polynomials of degree 1, 3, and 9, and then see bias and variance in numbers: underfitting is wrong on both train and test, while overfitting has a training error near 0 but a test error that shoots up.

1. Generating data with a curved pattern

import numpy as np

rng = np.random.default_rng(2)          # fixed seed → everyone gets the same results

def f(x):                               # true pattern: one period of a sine wave
    return np.sin(2 * np.pi * x)

x_train = np.linspace(0, 1, 10)
y_train = f(x_train) + rng.normal(0, 0.3, 10)     # 10 training samples, noise σ = 0.3
x_test = rng.uniform(0, 1, 200)
y_test = f(x_test) + rng.normal(0, 0.3, 200)      # 200 previously unseen samples

def mse(y, y_hat):
    return np.mean((y - y_hat) ** 2)

2. Fitting three models

for degree in (1, 3, 9):
    coef = np.polyfit(x_train, y_train, degree)
    tr = mse(y_train, np.polyval(coef, x_train))
    te = mse(y_test, np.polyval(coef, x_test))
    print(f"degree {degree}: train = {tr:.3f}   test = {te:.3f}")
# degree 1: train = 0.220   test = 0.353
# degree 3: train = 0.086   test = 0.146
# degree 9: train = 0.000   test = 0.914

With this data:

DegreeTrainTestDiagnosis
10.2200.353Underfitting (high bias): a straight line cannot bend along the sine wave, so it is wrong on both sets
30.0860.146Reasonable: the test error is close to the noise level σ² = 0.09, which no model can remove
90.0000.914Overfitting (high variance): 10 coefficients for exactly 10 points, so the curve passes through the noise as well

3. Sweeping every degree from 1 to 9

train_err, test_err = [], []
for degree in range(1, 10):
    coef = np.polyfit(x_train, y_train, degree)
    train_err.append(round(float(mse(y_train, np.polyval(coef, x_train))), 3))
    test_err.append(round(float(mse(y_test, np.polyval(coef, x_test))), 3))
print(train_err)   # [0.22, 0.217, 0.086, 0.081, 0.077, 0.06, 0.06, 0.009, 0.0]
print(test_err)    # [0.353, 0.359, 0.146, 0.154, 0.143, 0.158, 0.158, 0.501, 0.914]

The training error never increases as the degree increases (a higher-degree model contains the lower-degree models, so it fits train at least as well). The test error decreases and then rises again. In this run, the lowest test error is at degree 5, but it is almost the same as degree 3, so we choose degree 3 (simpler). This is specific data; running with a different seed will differ slightly.

4. Adding data reduces variance

for n in (10, 30, 100):
    x = np.linspace(0, 1, n)
    y = f(x) + rng.normal(0, 0.3, n)
    coef = np.polyfit(x, y, 9)
    print(n, round(mse(y, np.polyval(coef, x)), 3), round(mse(y_test, np.polyval(coef, x_test)), 3))
# 10 0.0 0.872
# 30 0.087 0.146
# 100 0.061 0.109

The degree-9 model does not change, but the more training samples there are, the lower the test error becomes (the first row differs from 0.914 because this is a fresh draw of noise). The model can no longer "memorize the noise" so easily.

Common mistakes

  • Looking only at the training error and choosing the most complex model.
  • Choosing the degree using the test set itself and then reporting that same test set (use cross-validation, lesson 3.7).

Key takeaways

  • Both train and test high: high bias. Train low, test high: high variance.
  • The train-test gap is the signal, not the training error alone.
  • Adding data helps reduce variance; adding complexity helps reduce bias.

Flashcards · 1.12 Lab: Bias-Variance Through Experiments

10 words
WordMeaningExample
  • np.polyfit()

    NumPy function that fits a polynomial of a given degree and returns its coefficients.

    coef = np.polyfit(x_train, y_train, 3)

  • np.polyval()

    Evaluates a polynomial with given coefficients at chosen x values to get predictions.

    y_hat = np.polyval(coef, x_test)

  • Polynomial degree

    Controls model complexity: higher degree means a more flexible curve with more coefficients.

    Degree 9 has 10 coefficients.

  • Underfitting

    A model too simple for the pattern; error is high on both training and test data (high bias).

    Degree 1 on a sine wave: train 0.220, test 0.353.

  • Overfitting

    A model so flexible it fits the noise; training error near 0 but test error high (high variance).

    Degree 9 on 10 points: train 0.000, test 0.914.

  • Noise level

    The irreducible error from random noise; no model's test MSE can go meaningfully below it.

    Noise σ = 0.3 gives σ² = 0.09.

  • Train-test gap

    The difference between training and test error; a large gap is the signal of high variance.

    Train 0.000 vs test 0.914.

  • More data reduces variance

    With more training samples, a flexible model can no longer memorise the noise so easily.

    Degree 9 with 100 points: test MSE about 0.109.

  • np.random.default_rng(seed)

    Creates a random generator with a fixed seed so everyone gets the same synthetic data.

    rng = np.random.default_rng(2)

  • Choose the simpler model

    When two models have almost the same test error, prefer the less complex one.

    Degree 3 over degree 5 at similar test error.

ML_Lab_1.12_notebook.zip· 2,1 KB