You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
A model that memorizes the entire training set will reach 100% accuracy on that set — and may be useless on new data. This is overfitting. To find out whether a model has truly learned the underlying pattern, you must test it on data it has never seen.
| Set | Typical proportion | Used for |
|---|---|---|
| Train | 60–80% | Training the model |
| Validation | 10–20% | Choosing the model, tuning hyperparameters |
| Test | 10–20% | Final evaluation, used only once |
If you only split into two parts (train/test) as in the lab session, use cross-validation on the training set to take over the role of the validation set.
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(
x, y, test_size=0.2, random_state=1009)
random_state fixes how the random split is made. Thanks to it, rerunning the code produces exactly the same result as before — a mandatory condition for comparing two alternatives fairly.
If only 1% of the data is fraud, a single random split can push almost all fraud cases into the training set, leaving the test set with almost no positive cases. The recall you measure is then meaningless.
x_train, x_test, y_train, y_test = train_test_split(
x, y, test_size=0.2, random_state=1009, stratify=y)
stratify=y preserves the class proportions in both sets.
Using the test set many times to tune the model. Every time you look at the test result and then go back to adjust the model, the test set leaks a little more into your decisions. After ten rounds like this, the number on the test set is no longer an honest estimate. Tune on the validation set, and touch the test set exactly once at the very end.
By the end of this lesson you can:
train_test_split with test_size, random_state and stratify.Key idea: Evaluate a model only on data it has never seen. Fix
random_stateso splits are reproducible, usestratify=ywhen classes are imbalanced, and touch the test set exactly once at the very end.
A model that memorises the entire training set can reach 100% accuracy on it and still be useless on new data. This is overfitting. To know whether a model has learned the real underlying pattern rather than the particular examples, you must test it on data it has never seen.
| Set | Typical share | Used for |
|---|---|---|
| Train | 60 to 80% | Training the model |
| Validation | 10 to 20% | Choosing the model and tuning hyperparameters |
| Test | 10 to 20% | Final evaluation, used only once |
If you split into only two parts (train and test), as in the lab sessions, use cross-validation on the training set to play the role of the validation set.
In scikit-learn the split is done with train_test_split(x, y, test_size=0.2, random_state=1009), which returns x_train, x_test, y_train, y_test in that order. With test_size=0.2, 20% of the rows go to the test set and 80% to training.
The split is random. random_state fixes the random choice, so rerunning the code gives exactly the same split. This is a requirement for comparing two alternatives fairly: if each run used a different split, you could not tell whether a better score came from a better model or a luckier split.
If only 1% of the rows are fraud, a single random split may put almost all the fraud cases into training and leave the test set with almost none. Any recall you measure on that test set is then meaningless. Adding stratify=y keeps the class proportions the same in both sets. For imbalanced classification it is mandatory, and for balanced data it does no harm.
Using the test set many times while tuning. Each time you look at the test result and go back to adjust the model, a little information from the test set leaks into your decisions. After ten such rounds the test score is no longer an honest estimate. Tune on the validation set (or with cross-validation) and use the test set exactly once at the end.
A dataset has 1,000 transactions, of which 10 (1%) are fraud. Compare a plain split with a stratified split.
import numpy as np
from sklearn.model_selection import train_test_split
x = np.arange(1000).reshape(-1, 1) # one dummy feature
y = np.array([1] * 10 + [0] * 990) # 10 fraud cases, 990 normal
x_train, x_test, y_train, y_test = train_test_split(
x, y, test_size=0.2, random_state=1009)
print("plain: ", len(y_train), len(y_test), y_train.sum(), y_test.sum())
x_train, x_test, y_train, y_test = train_test_split(
x, y, test_size=0.2, random_state=1009, stratify=y)
print("stratified:", len(y_train), len(y_test), y_train.sum(), y_test.sum())
Expected output: both lines show 800 training rows and 200 test rows. On the stratified line the fraud cases are split exactly in proportion: 8 in train and 2 in test (1% of each set). On the plain line the fraud count in the test set depends on chance; it may be 2, but it can easily be 0, 1, 3 or more. With 0 fraud cases in the test set you could not measure recall at all. Because random_state=1009 is fixed, rerunning the script always prints the same numbers.
random_state. Every run gets a different split, so comparisons between models are unfair.stratify=y with rare classes. The test set may contain too few positive cases for meaningful metrics.train_test_split returns x_train, x_test, y_train, y_test; test_size sets the test share.random_state makes the split reproducible; stratify=y preserves class proportions.Flashcards · Data Partitioning
8 wordsWhen a model memorises the training data, scoring very well on it but poorly on new data.
scikit-learn function that randomly divides features and target into training and test parts.
Argument of train_test_split giving the share (or number) of rows placed in the test set.
Fixes the random split so rerunning the code produces exactly the same sets.
Argument that keeps the class proportions the same in the training and test sets.
A classification dataset where one class is much rarer than the others.
Checking the test score repeatedly while tuning, which leaks information and inflates the final estimate.
Data held out from training and used to choose models and tune hyperparameters.