You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
We give the model pairs of (input, correct answer) and let it find the mapping pattern by itself. "Supervised" means every training sample comes with a correct label.
| Problem type | Output | Example |
|---|---|---|
| Regression | Continuous number | House price, tomorrow's temperature |
| Classification | Discrete label | Spam / not spam, sick / not sick |
High bias (underfitting) — the model is too simple to capture the pattern. Fitting a straight line to a curved relationship gives errors on both the training and test sets.
High variance (overfitting) — the model is too complex and memorizes even the noise in the training set. Nearly perfect on the training set, poor on the test set.
| Symptom | Train error | Test error | Diagnosis |
|---|---|---|---|
| High | High | High | Underfitting (high bias) |
| Large gap | Very low | High | Overfitting (high variance) |
| Good | Low | Low, close to train | Just right |
As model complexity increases, bias decreases but variance increases. The goal is not to eliminate either one, but to find the balance point that minimizes the total error on new data.
If the model is overfitting: add data, reduce the number of features, reduce complexity (for example, limit the max_depth of a tree), or add regularization.
If the model is underfitting: add features, use a more powerful model, or train for longer.
Seeing high accuracy on the training set and concluding the model is good. That number almost always looks great, even when the model has merely memorized the data. Always compare train with test: the gap between the two numbers is what reveals the problem.
By the end of this lesson you can:
Key idea: A more complex model is not automatically better. Too simple means high bias and errors everywhere; too complex means high variance and memorised noise. Compare training error with test error: the gap between them reveals the problem.
In supervised learning we give the model pairs of (input, correct answer) and let it discover the mapping by itself. "Supervised" means every training sample comes with a correct label.
| Problem type | Output | Examples |
|---|---|---|
| Regression | A continuous number | House price, tomorrow's temperature |
| Classification | A discrete label | Spam or not spam, sick or not sick |
High bias (underfitting): the model is too simple to capture the pattern. Fitting a straight line to a curved relationship produces errors on both the training set and the test set, because the line cannot bend.
High variance (overfitting): the model is too complex and memorises even the random noise in the training set. It is nearly perfect on training data and poor on test data.
| Situation | Train error | Test error | Diagnosis |
|---|---|---|---|
| Both high | High | High | Underfitting (high bias) |
| Large gap | Very low | High | Overfitting (high variance) |
| Good | Low | Low and close to train | Just right |
As model complexity increases, bias goes down but variance goes up. The goal is not to remove one of them completely; it is to find the balance point that gives the lowest total error on new data. That point is usually somewhere in the middle of the complexity range.
| Problem | Remedies |
|---|---|
| Overfitting | Add more data; use fewer features; reduce complexity (for example limit a tree's max_depth); add regularization |
| Underfitting | Add useful features; use a more powerful model; train for longer |
Notice that the remedies point in opposite directions. Before you act, diagnose which problem you have from the train and test errors.
Fit decision trees of increasing depth to noisy data with a curved (sine) pattern, and compare training and test error.
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error
rng = np.random.default_rng(0)
x_train = rng.uniform(0, 1, 40).reshape(-1, 1)
y_train = np.sin(2 * np.pi * x_train).ravel() + rng.normal(0, 0.3, 40)
x_test = rng.uniform(0, 1, 200).reshape(-1, 1)
y_test = np.sin(2 * np.pi * x_test).ravel() + rng.normal(0, 0.3, 200)
for depth in (1, 3, None): # None = no depth limit
tree = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(x_train, y_train)
train_mse = mean_squared_error(y_train, tree.predict(x_train))
test_mse = mean_squared_error(y_test, tree.predict(x_test))
print(f"max_depth={depth}: train={train_mse:.3f} test={test_mse:.3f}")
Expected pattern (exact numbers depend on the random noise):
max_depth=1: one split cannot follow a sine wave, so training and test errors are both high and similar. This is underfitting.max_depth=3: both errors drop and stay close to each other. This is near the balance point.max_depth=None: the tree grows until every training point has its own leaf, so the training error is 0.000. The test error, however, is far above the training error and typically higher than at depth 3, because the tree has memorised the noise. This is overfitting.Choosing by training error alone would pick the unlimited tree, the worst choice for new data.
Flashcards · Supervised Learning Foundations and the Bias-Variance Tradeoff
9 wordsLearning a mapping from inputs to outputs using training samples that each carry the correct label.
Error caused by a model being too simple to capture the real pattern.
Error caused by a model being so flexible that it changes a lot with the training data and fits noise.
High-bias situation where the model has high error on both training and test data.
High-variance situation where training error is very low but test error is high.
As complexity rises, bias falls and variance rises; the best model minimises total error on new data.
Decision tree setting that limits how many levels of splits the tree can grow, controlling complexity.
A technique that penalises model complexity to reduce overfitting.
The difference between training and test error; a large gap signals overfitting.