You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
A single train/test split gives you one number. But that number depends on where the data happens to land. Change random_state and the result can jump from 0.74 to 0.81 — even though the model has not changed at all. So which number is the real one?
With K = 5: split the data into 5 equal parts, then repeat 5 times — each time take one part as the validation set and use the other 4 parts for training. Finally, average the 5 results.
Every data sample is used for validation exactly once, and for training K−1 times.
from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipe, x_train, y_train, cv=5, scoring="f1")
print(scores) # 5 numbers
print(scores.mean(), scores.std()) # the mean and the spread
The standard deviation matters as much as the mean. A mean of 0.80 with a standard deviation of 0.02 is a stable model. A mean of 0.80 with a standard deviation of 0.15 means the result depends heavily on how the data is split — it is not yet trustworthy.
For classification problems, always use the stratified version so that each fold keeps the correct class proportions. Scikit-learn does this automatically when you pass a classifier to cross_val_score.
| K | Tradeoff |
|---|---|
| 5 | Fast and usually good enough — a reasonable default |
| 10 | More stable estimate, twice as slow |
| = number of samples (LOO) | Least biased but very slow, only suitable for small datasets |
Normalizing the data before running cross-validation. The scaler has then learned from the entire dataset, including the part that will serve as the validation set in each round — so the result looks better than reality. How to avoid it: wrap the scaler and the model in a Pipeline, then pass that pipeline to cross_val_score.
By the end of this lesson you can:
PipelineKey idea: Cross-validation trains and evaluates the model K times on different splits and reports the mean and spread of the scores, giving a far more reliable estimate than one lucky or unlucky split.
A single train/test split gives you one number, and that number depends on where the data happens to land. Change random_state and the score can jump from 0.74 to 0.81 although the model has not changed at all. So which number is the real one? Neither, on its own.
With K = 5:
Every sample is used for validation exactly once and for training K - 1 times. With 2,000 rows and 5 folds, each round trains on 1,600 rows and validates on 400.
from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipe, x_train, y_train, cv=5, scoring="f1")
print(scores) # 5 numbers
print(scores.mean(), scores.std()) # the mean and the spread
For classification, always use the stratified version, so that each fold keeps the correct class proportions. Otherwise, with imbalanced data, one fold could contain almost no minority samples. scikit-learn does this automatically when you pass a classifier and an integer cv to cross_val_score.
| K | Trade-off |
|---|---|
| 5 | fast and usually good enough: a reasonable default |
| 10 | more stable estimate, about twice as slow |
| number of samples (leave-one-out, LOO) | least biased but very slow; only for small datasets |
If you normalise the whole dataset first and then run cross-validation, the scaler has learned the mean and standard deviation from all the data, including each round's validation fold. Information leaks from validation into training, so the result looks better than reality.
The fix: wrap the scaler and the model in a Pipeline and pass the pipeline to cross_val_score. Then, in every round, the scaler is fitted on the training folds only.
Evaluate an SVM honestly with a pipeline.
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
pipe = Pipeline([
("scaler", StandardScaler()),
("model", SVC()),
])
scores = cross_val_score(pipe, x_train, y_train, cv=5, scoring="f1")
print(scores.round(3))
print(f"{scores.mean():.3f} +/- {scores.std():.3f}")
Reasoning step by step:
cv=5 with a classifier means stratified 5-fold: each fold keeps the class proportions.StandardScaler on 4 folds only, transforms the 5th, trains the SVM, and computes F1 on the 5th fold.0.71 +/- 0.03 (the exact values depend on your data).Pipeline.random_state.Pipeline so it is fitted on the training folds only.Flashcards · Cross-Validation
8 wordsEvaluating a model several times on different train/validation splits and combining the scores.
Split data into K folds; each round validates on one fold and trains on the other K - 1.
One of the K equal parts the data is divided into for cross-validation.
The spread of fold scores; a large spread means the estimate depends heavily on the split.
K-fold that keeps the class proportions in every fold; automatic for classifiers in cross_val_score.
Cross-validation with K equal to the number of samples; least biased but very slow.
Fitting preprocessing on all data before CV lets validation information into training, inflating scores.
A scikit-learn object chaining preprocessing and a model so each step is fitted on training data only.