You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
After this lab you will be able to: see for yourself how KFold splits the data, read the results of cross_val_score, and use GridSearchCV to choose hyperparameters (and count the number of training runs) without touching the test set.
import numpy as np
from sklearn.model_selection import KFold
X10 = np.arange(10).reshape(-1, 1) # 10 samples, indices 0..9
for i, (tr, va) in enumerate(KFold(n_splits=5).split(X10)):
print(i, "train", tr, "val", va)
# 0 train [2 3 4 5 6 7 8 9] val [0 1]
# 1 train [0 1 4 5 6 7 8 9] val [2 3]
# ... (last fold: train [0..7], val [8 9])
print([va.tolist() for _, va in KFold(n_splits=3).split(X10)])
# [[0, 1, 2, 3], [4, 5, 6], [7, 8, 9]]
Each sample is in the validation set exactly once and in the training set K − 1 times. 10 samples do not divide evenly into 3, so the folds have sizes 4, 3, 3. KFold does not shuffle: it cuts the data in its existing order.
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
X, y = load_iris(return_X_y=True) # 150 samples, sorted by species: 50 of species 0, then 50 of species 1, then 50 of species 2
model = LogisticRegression(max_iter=1000)
scores = cross_val_score(model, X, y, cv=5)
print(len(scores)) # 5 (one score per fold)
print(f"{scores.mean():.3f} ± {scores.std():.3f}") # 0.973 ± 0.025
The mean is the estimate of the expected performance; the standard deviation shows how much the result fluctuates across different splits (the exact numbers may differ slightly between versions). Be careful with data sorted by label:
print(cross_val_score(model, X, y, cv=KFold(n_splits=3))) # [0. 0. 0.]
shuffled = KFold(n_splits=3, shuffle=True, random_state=0)
print(cross_val_score(model, X, y, cv=shuffled).min() > 0.9) # True
Without shuffling, each validation fold contains an entire species that the two training folds have never seen, so the score is 0. When you pass cv=5 for a classifier, scikit-learn uses StratifiedKFold, which does not have this problem.
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeClassifier
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42) # 120 train / 30 test
grid = {"max_depth": [1, 2, 3, None], "min_samples_leaf": [1, 5]}
search = GridSearchCV(DecisionTreeClassifier(random_state=0), grid, cv=5, verbose=1)
search.fit(X_train, y_train) # use the training set only
# Fitting 5 folds for each of 8 candidates, totalling 40 fits
print(len(search.cv_results_["params"])) # 8
print(search.best_params_) # {'max_depth': None, 'min_samples_leaf': 1}
print(round(search.best_score_, 3)) # 0.942
print(round(search.score(X_test, y_test), 3)) # 0.967 ← the first and only time the test set is used
refit run.best_params_ / best_score_: the combination with the highest mean CV score, and that score (computed on the training set only); the exact values may differ between versions.refit=True (the default): after selecting, it retrains the best combination on the entire training set, and the result is stored in best_estimator_; search.score/search.predict use this model.The correct workflow: split train/test → GridSearchCV on the training set only → finalize the model → score on the test set exactly once.
Pipeline (the parameter then becomes step_name__max_depth).cross_val_score returns K scores: look at both the mean and the std.GridSearchCV: number of training runs = number of combinations × number of folds (+1 refit).Flashcards · 3.7 Lab: Cross-Validation and GridSearchCV
10 wordsSplits data into K folds; each sample is validated once and trained on K-1 times. It does not shuffle by default.
When samples do not divide evenly, KFold makes the first folds one sample larger.
Trains and scores a model on each fold, returning one score per fold.
The mean estimates expected performance; the standard deviation shows how much results vary between splits.
Data sorted by label needs shuffled or stratified folds, otherwise a fold may contain a class never seen in training.
Fold splitter that keeps class proportions in each fold; used automatically when cv=5 is given for a classifier.
Tries every hyperparameter combination with cross-validation on the training set and keeps the best one.
GridSearchCV trains combinations x folds models, plus one refit of the best combination.
Default GridSearchCV setting that retrains the best combination on the whole training set as best_estimator_.
Choose hyperparameters on training data only; score the final model on the test set exactly once.