Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 3: Model Evaluation and Optimization

3.7 Lab: Cross-Validation and GridSearchCV

Print / Save as PDF

Lab: Cross-Validation and GridSearchCV

After this lab you will be able to: see for yourself how KFold splits the data, read the results of cross_val_score, and use GridSearchCV to choose hyperparameters (and count the number of training runs) without touching the test set.

1. How KFold splits the data

import numpy as np
from sklearn.model_selection import KFold

X10 = np.arange(10).reshape(-1, 1)             # 10 samples, indices 0..9
for i, (tr, va) in enumerate(KFold(n_splits=5).split(X10)):
    print(i, "train", tr, "val", va)
# 0 train [2 3 4 5 6 7 8 9] val [0 1]
# 1 train [0 1 4 5 6 7 8 9] val [2 3]
# ...  (last fold: train [0..7], val [8 9])

print([va.tolist() for _, va in KFold(n_splits=3).split(X10)])
# [[0, 1, 2, 3], [4, 5, 6], [7, 8, 9]]

Each sample is in the validation set exactly once and in the training set K − 1 times. 10 samples do not divide evenly into 3, so the folds have sizes 4, 3, 3. KFold does not shuffle: it cuts the data in its existing order.

2. cross_val_score: mean ± standard deviation

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

X, y = load_iris(return_X_y=True)   # 150 samples, sorted by species: 50 of species 0, then 50 of species 1, then 50 of species 2
model = LogisticRegression(max_iter=1000)

scores = cross_val_score(model, X, y, cv=5)
print(len(scores))                                # 5  (one score per fold)
print(f"{scores.mean():.3f} ± {scores.std():.3f}")   # 0.973 ± 0.025

The mean is the estimate of the expected performance; the standard deviation shows how much the result fluctuates across different splits (the exact numbers may differ slightly between versions). Be careful with data sorted by label:

print(cross_val_score(model, X, y, cv=KFold(n_splits=3)))    # [0. 0. 0.]
shuffled = KFold(n_splits=3, shuffle=True, random_state=0)
print(cross_val_score(model, X, y, cv=shuffled).min() > 0.9)  # True

Without shuffling, each validation fold contains an entire species that the two training folds have never seen, so the score is 0. When you pass cv=5 for a classifier, scikit-learn uses StratifiedKFold, which does not have this problem.

3. GridSearchCV and the rule of not using the test set

from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeClassifier

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)     # 120 train / 30 test

grid = {"max_depth": [1, 2, 3, None], "min_samples_leaf": [1, 5]}
search = GridSearchCV(DecisionTreeClassifier(random_state=0), grid, cv=5, verbose=1)
search.fit(X_train, y_train)            # use the training set only
# Fitting 5 folds for each of 8 candidates, totalling 40 fits

print(len(search.cv_results_["params"]))     # 8
print(search.best_params_)                   # {'max_depth': None, 'min_samples_leaf': 1}
print(round(search.best_score_, 3))          # 0.942
print(round(search.score(X_test, y_test), 3))  # 0.967  ← the first and only time the test set is used
  • Number of training runs = number of combinations × number of folds = 4 × 2 × 5 = 40, plus 1 refit run.
  • best_params_ / best_score_: the combination with the highest mean CV score, and that score (computed on the training set only); the exact values may differ between versions.
  • refit=True (the default): after selecting, it retrains the best combination on the entire training set, and the result is stored in best_estimator_; search.score/search.predict use this model.

The correct workflow: split train/test → GridSearchCV on the training set only → finalize the model → score on the test set exactly once.

Common Mistakes

  • Choosing hyperparameters by their score on the test set and then reporting that same score.
  • Standardizing the whole table before CV: put the scaler inside a Pipeline (the parameter then becomes step_name__max_depth).

Key Points

  • cross_val_score returns K scores: look at both the mean and the std.
  • GridSearchCV: number of training runs = number of combinations × number of folds (+1 refit).
  • The test set is used only once, at the final step.

Flashcards · 3.7 Lab: Cross-Validation and GridSearchCV

10 words
WordMeaningExample
  • KFold

    Splits data into K folds; each sample is validated once and trained on K-1 times. It does not shuffle by default.

    KFold(n_splits=5).split(X10)

  • Fold sizes

    When samples do not divide evenly, KFold makes the first folds one sample larger.

    10 samples, 3 folds -> sizes 4, 3, 3.

  • cross_val_score()

    Trains and scores a model on each fold, returning one score per fold.

    cross_val_score(model, X, y, cv=5) -> 5 scores

  • Mean and standard deviation of CV scores

    The mean estimates expected performance; the standard deviation shows how much results vary between splits.

    0.973 +/- 0.025

  • Shuffling with sorted labels

    Data sorted by label needs shuffled or stratified folds, otherwise a fold may contain a class never seen in training.

    KFold(n_splits=3, shuffle=True, random_state=0)

  • StratifiedKFold

    Fold splitter that keeps class proportions in each fold; used automatically when cv=5 is given for a classifier.

    cross_val_score(clf, X, y, cv=5)

  • GridSearchCV

    Tries every hyperparameter combination with cross-validation on the training set and keeps the best one.

    GridSearchCV(DecisionTreeClassifier(), grid, cv=5)

  • Number of fits

    GridSearchCV trains combinations x folds models, plus one refit of the best combination.

    4 x 2 combinations x 5 folds = 40 fits, + 1 refit.

  • refit=True

    Default GridSearchCV setting that retrains the best combination on the whole training set as best_estimator_.

    search.best_estimator_.predict(X_test)

  • Test set used once

    Choose hyperparameters on training data only; score the final model on the test set exactly once.

    search.score(X_test, y_test) at the very end.

ML_Lab_3.7_notebook.zip· 2,4 KB