Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1: Data Preparation and Model Foundations

Data Partitioning

Print / Save as PDF

Data Splitting

Why you must not evaluate on the training data

A model that memorizes the entire training set will reach 100% accuracy on that set — and may be useless on new data. This is overfitting. To find out whether a model has truly learned the underlying pattern, you must test it on data it has never seen.

How to split

SetTypical proportionUsed for
Train60–80%Training the model
Validation10–20%Choosing the model, tuning hyperparameters
Test10–20%Final evaluation, used only once

If you only split into two parts (train/test) as in the lab session, use cross-validation on the training set to take over the role of the validation set.

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.2, random_state=1009)

random_state fixes how the random split is made. Thanks to it, rerunning the code produces exactly the same result as before — a mandatory condition for comparing two alternatives fairly.

Stratified splitting (stratify) — mandatory for imbalanced data

If only 1% of the data is fraud, a single random split can push almost all fraud cases into the training set, leaving the test set with almost no positive cases. The recall you measure is then meaningless.

x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.2, random_state=1009, stratify=y)

stratify=y preserves the class proportions in both sets.

The most common mistake

Using the test set many times to tune the model. Every time you look at the test result and then go back to adjust the model, the test set leaks a little more into your decisions. After ten rounds like this, the number on the test set is no longer an honest estimate. Tune on the validation set, and touch the test set exactly once at the very end.

Study notes

By the end of this lesson you can:

  • Explain why a model must be evaluated on data it has never seen.
  • Choose sensible proportions for train, validation and test sets.
  • Use train_test_split with test_size, random_state and stratify.
  • Explain why the test set must be used only once.

Key idea: Evaluate a model only on data it has never seen. Fix random_state so splits are reproducible, use stratify=y when classes are imbalanced, and touch the test set exactly once at the very end.

Why you must not evaluate on the training data

A model that memorises the entire training set can reach 100% accuracy on it and still be useless on new data. This is overfitting. To know whether a model has learned the real underlying pattern rather than the particular examples, you must test it on data it has never seen.

How to split

SetTypical shareUsed for
Train60 to 80%Training the model
Validation10 to 20%Choosing the model and tuning hyperparameters
Test10 to 20%Final evaluation, used only once

If you split into only two parts (train and test), as in the lab sessions, use cross-validation on the training set to play the role of the validation set.

In scikit-learn the split is done with train_test_split(x, y, test_size=0.2, random_state=1009), which returns x_train, x_test, y_train, y_test in that order. With test_size=0.2, 20% of the rows go to the test set and 80% to training.

random_state: making splits reproducible

The split is random. random_state fixes the random choice, so rerunning the code gives exactly the same split. This is a requirement for comparing two alternatives fairly: if each run used a different split, you could not tell whether a better score came from a better model or a luckier split.

Stratified splitting for imbalanced data

If only 1% of the rows are fraud, a single random split may put almost all the fraud cases into training and leave the test set with almost none. Any recall you measure on that test set is then meaningless. Adding stratify=y keeps the class proportions the same in both sets. For imbalanced classification it is mandatory, and for balanced data it does no harm.

The most common mistake

Using the test set many times while tuning. Each time you look at the test result and go back to adjust the model, a little information from the test set leaks into your decisions. After ten such rounds the test score is no longer an honest estimate. Tune on the validation set (or with cross-validation) and use the test set exactly once at the end.

Worked example

A dataset has 1,000 transactions, of which 10 (1%) are fraud. Compare a plain split with a stratified split.

import numpy as np
from sklearn.model_selection import train_test_split

x = np.arange(1000).reshape(-1, 1)        # one dummy feature
y = np.array([1] * 10 + [0] * 990)        # 10 fraud cases, 990 normal

x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.2, random_state=1009)
print("plain:     ", len(y_train), len(y_test), y_train.sum(), y_test.sum())

x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.2, random_state=1009, stratify=y)
print("stratified:", len(y_train), len(y_test), y_train.sum(), y_test.sum())

Expected output: both lines show 800 training rows and 200 test rows. On the stratified line the fraud cases are split exactly in proportion: 8 in train and 2 in test (1% of each set). On the plain line the fraud count in the test set depends on chance; it may be 2, but it can easily be 0, 1, 3 or more. With 0 fraud cases in the test set you could not measure recall at all. Because random_state=1009 is fixed, rerunning the script always prints the same numbers.

Common mistakes

  • Reporting accuracy on the training set. It rewards memorisation; always report results on unseen data.
  • Leaving out random_state. Every run gets a different split, so comparisons between models are unfair.
  • Forgetting stratify=y with rare classes. The test set may contain too few positive cases for meaningful metrics.
  • Tuning against the test set. Repeated peeking turns the test set into a second validation set and inflates the final score.

Summary

  • A model must be judged on data it has never seen; training accuracy can hide overfitting.
  • Typical shares: train 60 to 80%, validation 10 to 20%, test 10 to 20%.
  • train_test_split returns x_train, x_test, y_train, y_test; test_size sets the test share.
  • random_state makes the split reproducible; stratify=y preserves class proportions.
  • Tune on validation data and use the test set once.

Flashcards · Data Partitioning

8 words
WordMeaningExample
  • Overfitting

    When a model memorises the training data, scoring very well on it but poorly on new data.

    100% training accuracy, 60% test accuracy.

  • train_test_split()

    scikit-learn function that randomly divides features and target into training and test parts.

    train_test_split(x, y, test_size=0.2, random_state=1009)

  • test_size

    Argument of train_test_split giving the share (or number) of rows placed in the test set.

    test_size=0.3 on 500 rows gives 150 test rows.

  • random_state

    Fixes the random split so rerunning the code produces exactly the same sets.

    random_state=1009

  • stratify

    Argument that keeps the class proportions the same in the training and test sets.

    train_test_split(x, y, stratify=y)

  • Imbalanced data

    A classification dataset where one class is much rarer than the others.

    1% fraud, 99% normal transactions.

  • Test set reuse

    Checking the test score repeatedly while tuning, which leaks information and inflates the final estimate.

    Ten rounds of tweak, test, tweak.

  • Validation set

    Data held out from training and used to choose models and tune hyperparameters.

    10-20% of the data, or cross-validation folds.