Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 3: Model Evaluation and Optimization

Cross-Validation

Print / Save as PDF

Cross-Validation

The Problem with a Single Split

A single train/test split gives you one number. But that number depends on where the data happens to land. Change random_state and the result can jump from 0.74 to 0.81 — even though the model has not changed at all. So which number is the real one?

K-Fold: Split into K Parts and Rotate

With K = 5: split the data into 5 equal parts, then repeat 5 times — each time take one part as the validation set and use the other 4 parts for training. Finally, average the 5 results.

Every data sample is used for validation exactly once, and for training K−1 times.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipe, x_train, y_train, cv=5, scoring="f1")
print(scores)                        # 5 numbers
print(scores.mean(), scores.std())   # the mean and the spread

The standard deviation matters as much as the mean. A mean of 0.80 with a standard deviation of 0.02 is a stable model. A mean of 0.80 with a standard deviation of 0.15 means the result depends heavily on how the data is split — it is not yet trustworthy.

Stratified K-Fold

For classification problems, always use the stratified version so that each fold keeps the correct class proportions. Scikit-learn does this automatically when you pass a classifier to cross_val_score.

Choosing K

KTradeoff
5Fast and usually good enough — a reasonable default
10More stable estimate, twice as slow
= number of samples (LOO)Least biased but very slow, only suitable for small datasets

The Most Common Mistake

Normalizing the data before running cross-validation. The scaler has then learned from the entire dataset, including the part that will serve as the validation set in each round — so the result looks better than reality. How to avoid it: wrap the scaler and the model in a Pipeline, then pass that pipeline to cross_val_score.

Study notes

By the end of this lesson you can:

  • Explain why a single train/test split gives an unreliable score
  • Describe K-fold cross-validation and how each sample is used
  • Read the mean and the standard deviation of cross-validation scores
  • Explain stratified K-fold and choose a sensible value of K
  • Avoid data leakage by putting the scaler inside a Pipeline

Key idea: Cross-validation trains and evaluates the model K times on different splits and reports the mean and spread of the scores, giving a far more reliable estimate than one lucky or unlucky split.

The problem with a single split

A single train/test split gives you one number, and that number depends on where the data happens to land. Change random_state and the score can jump from 0.74 to 0.81 although the model has not changed at all. So which number is the real one? Neither, on its own.

K-fold: split into K parts and rotate

With K = 5:

  1. split the data into 5 equal parts (folds);
  2. repeat 5 times: use one fold as the validation set and the other 4 for training;
  3. average the 5 results.

Every sample is used for validation exactly once and for training K - 1 times. With 2,000 rows and 5 folds, each round trains on 1,600 rows and validates on 400.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipe, x_train, y_train, cv=5, scoring="f1")
print(scores)                        # 5 numbers
print(scores.mean(), scores.std())   # the mean and the spread

The standard deviation matters as much as the mean

  • Mean 0.80, standard deviation 0.02: a stable model; the score barely depends on the split.
  • Mean 0.80, standard deviation 0.15: the result depends heavily on how the data is split; it is not yet trustworthy. Collect more data, simplify the model, or investigate unusual folds.

Stratified K-fold

For classification, always use the stratified version, so that each fold keeps the correct class proportions. Otherwise, with imbalanced data, one fold could contain almost no minority samples. scikit-learn does this automatically when you pass a classifier and an integer cv to cross_val_score.

Choosing K

KTrade-off
5fast and usually good enough: a reasonable default
10more stable estimate, about twice as slow
number of samples (leave-one-out, LOO)least biased but very slow; only for small datasets

The most common mistake: scaling before cross-validation

If you normalise the whole dataset first and then run cross-validation, the scaler has learned the mean and standard deviation from all the data, including each round's validation fold. Information leaks from validation into training, so the result looks better than reality.

The fix: wrap the scaler and the model in a Pipeline and pass the pipeline to cross_val_score. Then, in every round, the scaler is fitted on the training folds only.

Worked example

Evaluate an SVM honestly with a pipeline.

from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", SVC()),
])

scores = cross_val_score(pipe, x_train, y_train, cv=5, scoring="f1")
print(scores.round(3))
print(f"{scores.mean():.3f} +/- {scores.std():.3f}")

Reasoning step by step:

  1. cv=5 with a classifier means stratified 5-fold: each fold keeps the class proportions.
  2. In each of the 5 rounds, the pipeline fits StandardScaler on 4 folds only, transforms the 5th, trains the SVM, and computes F1 on the 5th fold.
  3. The output is 5 F1 scores, then a line such as 0.71 +/- 0.03 (the exact values depend on your data).
  4. Interpretation: a small spread means the estimate is stable; a large spread means you should not trust a single number yet.
  5. The test set is not touched here; it is kept for the final evaluation.

Common mistakes

  • Normalising the data before cross-validation. The scaler sees the validation folds and the score is optimistic; use a Pipeline.
  • Looking only at the mean. The standard deviation tells you whether the estimate is trustworthy.
  • Using plain, unshuffled folds on sorted classification data. Use stratified folds so every fold has the right class mix.
  • Choosing K = number of samples on a large dataset. Leave-one-out is extremely slow; K = 5 or 10 is usually enough.

Summary

  • A single split gives one noisy number that changes with random_state.
  • K-fold rotates the validation fold; each sample is validated once and trained on K - 1 times.
  • Report the mean and the standard deviation of the K scores.
  • Use stratified folds for classification; K = 5 is a sensible default.
  • Put preprocessing inside a Pipeline so it is fitted on the training folds only.

Flashcards · Cross-Validation

8 words
WordMeaningExample
  • Cross-validation

    Evaluating a model several times on different train/validation splits and combining the scores.

    cross_val_score(pipe, x_train, y_train, cv=5)

  • K-fold

    Split data into K folds; each round validates on one fold and trains on the other K - 1.

    K = 5: 2,000 rows -> train on 1,600, validate on 400.

  • Fold

    One of the K equal parts the data is divided into for cross-validation.

    With cv=5, each fold holds 20% of the rows.

  • CV standard deviation

    The spread of fold scores; a large spread means the estimate depends heavily on the split.

    0.80 +/- 0.15 is not yet trustworthy.

  • Stratified K-fold

    K-fold that keeps the class proportions in every fold; automatic for classifiers in cross_val_score.

    StratifiedKFold(n_splits=5)

  • Leave-one-out (LOO)

    Cross-validation with K equal to the number of samples; least biased but very slow.

    Only for small datasets.

  • Data leakage in CV

    Fitting preprocessing on all data before CV lets validation information into training, inflating scores.

    StandardScaler fitted before cross_val_score.

  • Pipeline

    A scikit-learn object chaining preprocessing and a model so each step is fitted on training data only.

    Pipeline([('scaler', StandardScaler()), ('model', SVC())])