Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1: Data Preparation and Model Foundations

Supervised Learning Foundations and the Bias-Variance Tradeoff

Print / Save as PDF

Supervised Learning Foundations and the Bias-Variance Tradeoff

Supervised learning

We give the model pairs of (input, correct answer) and let it find the mapping pattern by itself. "Supervised" means every training sample comes with a correct label.

Problem typeOutputExample
RegressionContinuous numberHouse price, tomorrow's temperature
ClassificationDiscrete labelSpam / not spam, sick / not sick

Bias and Variance

High bias (underfitting) — the model is too simple to capture the pattern. Fitting a straight line to a curved relationship gives errors on both the training and test sets.

High variance (overfitting) — the model is too complex and memorizes even the noise in the training set. Nearly perfect on the training set, poor on the test set.

SymptomTrain errorTest errorDiagnosis
HighHighHighUnderfitting (high bias)
Large gapVery lowHighOverfitting (high variance)
GoodLowLow, close to trainJust right

The tradeoff

As model complexity increases, bias decreases but variance increases. The goal is not to eliminate either one, but to find the balance point that minimizes the total error on new data.

How to recognize and handle it

If the model is overfitting: add data, reduce the number of features, reduce complexity (for example, limit the max_depth of a tree), or add regularization.

If the model is underfitting: add features, use a more powerful model, or train for longer.

The most common mistake

Seeing high accuracy on the training set and concluding the model is good. That number almost always looks great, even when the model has merely memorized the data. Always compare train with test: the gap between the two numbers is what reveals the problem.

Study notes

By the end of this lesson you can:

  • Define supervised learning and give regression and classification examples.
  • Explain high bias (underfitting) and high variance (overfitting) in your own words.
  • Diagnose a model from its training error and test error.
  • Suggest remedies for overfitting and for underfitting.

Key idea: A more complex model is not automatically better. Too simple means high bias and errors everywhere; too complex means high variance and memorised noise. Compare training error with test error: the gap between them reveals the problem.

Supervised learning

In supervised learning we give the model pairs of (input, correct answer) and let it discover the mapping by itself. "Supervised" means every training sample comes with a correct label.

Problem typeOutputExamples
RegressionA continuous numberHouse price, tomorrow's temperature
ClassificationA discrete labelSpam or not spam, sick or not sick

Bias and variance

High bias (underfitting): the model is too simple to capture the pattern. Fitting a straight line to a curved relationship produces errors on both the training set and the test set, because the line cannot bend.

High variance (overfitting): the model is too complex and memorises even the random noise in the training set. It is nearly perfect on training data and poor on test data.

SituationTrain errorTest errorDiagnosis
Both highHighHighUnderfitting (high bias)
Large gapVery lowHighOverfitting (high variance)
GoodLowLow and close to trainJust right

The tradeoff

As model complexity increases, bias goes down but variance goes up. The goal is not to remove one of them completely; it is to find the balance point that gives the lowest total error on new data. That point is usually somewhere in the middle of the complexity range.

How to recognise and fix each problem

ProblemRemedies
OverfittingAdd more data; use fewer features; reduce complexity (for example limit a tree's max_depth); add regularization
UnderfittingAdd useful features; use a more powerful model; train for longer

Notice that the remedies point in opposite directions. Before you act, diagnose which problem you have from the train and test errors.

Worked example

Fit decision trees of increasing depth to noisy data with a curved (sine) pattern, and compare training and test error.

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(0)
x_train = rng.uniform(0, 1, 40).reshape(-1, 1)
y_train = np.sin(2 * np.pi * x_train).ravel() + rng.normal(0, 0.3, 40)
x_test = rng.uniform(0, 1, 200).reshape(-1, 1)
y_test = np.sin(2 * np.pi * x_test).ravel() + rng.normal(0, 0.3, 200)

for depth in (1, 3, None):          # None = no depth limit
    tree = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(x_train, y_train)
    train_mse = mean_squared_error(y_train, tree.predict(x_train))
    test_mse = mean_squared_error(y_test, tree.predict(x_test))
    print(f"max_depth={depth}: train={train_mse:.3f}  test={test_mse:.3f}")

Expected pattern (exact numbers depend on the random noise):

  • max_depth=1: one split cannot follow a sine wave, so training and test errors are both high and similar. This is underfitting.
  • max_depth=3: both errors drop and stay close to each other. This is near the balance point.
  • max_depth=None: the tree grows until every training point has its own leaf, so the training error is 0.000. The test error, however, is far above the training error and typically higher than at depth 3, because the tree has memorised the noise. This is overfitting.

Choosing by training error alone would pick the unlimited tree, the worst choice for new data.

Common mistakes

  • Judging a model by training accuracy. That number almost always looks great, even when the model has only memorised the data.
  • Assuming more complexity always helps. Past the balance point, extra complexity increases test error.
  • Applying the wrong remedy. More data mainly helps variance; it does little for a model that is simply too simple.
  • Ignoring the gap. Low train error with high test error is the signature of overfitting.

Summary

  • Supervised learning learns from (input, correct answer) pairs; regression predicts numbers, classification predicts labels.
  • High bias: errors high on both sets. High variance: low train error, high test error.
  • Increasing complexity lowers bias and raises variance; aim for the lowest error on new data.
  • Fix overfitting with more data, fewer features, less complexity or regularization.
  • Fix underfitting with more features, a more powerful model or longer training.

Flashcards · Supervised Learning Foundations and the Bias-Variance Tradeoff

9 words
WordMeaningExample
  • Supervised learning

    Learning a mapping from inputs to outputs using training samples that each carry the correct label.

    Learning house prices from past sales.

  • Bias

    Error caused by a model being too simple to capture the real pattern.

    A straight line fitted to a curved relationship.

  • Variance

    Error caused by a model being so flexible that it changes a lot with the training data and fits noise.

    A very deep tree that memorises every point.

  • Underfitting

    High-bias situation where the model has high error on both training and test data.

    Train MSE 0.40, test MSE 0.42.

  • Overfitting

    High-variance situation where training error is very low but test error is high.

    99% train accuracy, 62% test accuracy.

  • Bias-variance tradeoff

    As complexity rises, bias falls and variance rises; the best model minimises total error on new data.

    Choosing max_depth=3 over 1 or unlimited.

  • max_depth

    Decision tree setting that limits how many levels of splits the tree can grow, controlling complexity.

    DecisionTreeRegressor(max_depth=3)

  • Regularization

    A technique that penalises model complexity to reduce overfitting.

    Adding a penalty on large coefficients.

  • Train-test gap

    The difference between training and test error; a large gap signals overfitting.

    Train error 0.00 vs test error 0.30.