Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 3: Model Evaluation and Optimization

3.P4 Standardization with StandardScaler

Print / Save as PDF

3.P4 Standardization with StandardScaler

After this unit you will: standardize data correctly with StandardScaler without leaking test data.

Key points

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
x_train = scaler.fit_transform(x_train)   # fit ONLY on the training set
x_test = scaler.transform(x_test)         # the test set is only transformed
Formula
Standardization (z-score)z = (x − u) / s where u is the mean and s is the standard deviation
  • fit_transform learns the mean/std from the training set and then applies them to that same set; transform only applies the parameters that were already learned.
  • After standardization, x_train and x_test become NumPy arrays with shapes (614, 8) and (154, 8).
  • If you also fit on the test set, information from the test set leaks into the standardization step and the evaluation results look better than they really are.

Study notes

By the end of this lesson you can:

  • Explain why features with very different ranges need standardisation
  • Apply the formula z = (x - u) / s used by StandardScaler
  • Explain the difference between fit, transform and fit_transform
  • Standardise the training and test sets correctly without leaking test information
  • Predict the shapes and properties of the data after scaling

Key idea: Learn the mean and standard deviation from the training set only (fit_transform on train), then reuse exactly those numbers on the test set (transform on test).

Why standardise?

The eight diabetes features have very different ranges. For example, Pregnancies goes from 0 to 17, while Glucose goes up to about 200. As discussed in the theory part, when features have different ranges, a model can pay more attention to the features with larger values. To make the model see all inputs fairly, we put them on the same scale.

StandardScaler and its formula

scikit-learn's StandardScaler implements standardisation. Its documentation gives the standardised value of a sample x as:

z = (x - u) / s

where u is the mean of the column and s is its standard deviation. After this transformation, each column of the training data has mean 0 and standard deviation 1.

fit, transform and fit_transform

The two methods used in the notebook are fit_transform and transform. The difference:

MethodWhat it does
fitcalculates the mean and standard deviation of each column; no transformation yet
transformapplies z = (x - u) / s using the mean and standard deviation already calculated
fit_transformfit then transform in one call

So, for the Pregnancies column, fit computes its mean and standard deviation, and transform takes every value, subtracts that mean and divides by that standard deviation. The same is done for every column with its own mean and standard deviation.

Train with fit_transform, test with transform

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
x_train = scaler.fit_transform(x_train)   # fit ONLY on the training set
x_test = scaler.transform(x_test)         # the test set is only transformed
  • On the training set we use fit_transform: compute the statistics, then transform.
  • On the test set we do not recalculate anything; we only transform it with the mean and standard deviation learned from training.

Why? The test set represents new, unseen data. If we also fitted on the test set, information from it would leak into preprocessing and the evaluation would look better than it really is.

After scaling, x_train and x_test become NumPy arrays (no column names) with shapes (614, 8) and (154, 8). The values look completely different from the originals but are now on comparable scales. Because the test set uses the training statistics, its column means are close to 0, but not exactly 0.

Worked example

A tiny Glucose column shows exactly what is learned and reused.

import pandas as pd
from sklearn.preprocessing import StandardScaler

train = pd.DataFrame({"Glucose": [100, 120, 140]})
test = pd.DataFrame({"Glucose": [130]})

scaler = StandardScaler()
print(scaler.fit_transform(train).ravel().round(3))
print(scaler.mean_, scaler.scale_.round(2))
print(scaler.transform(test).round(3))

Expected output and reasoning:

  1. fit learns u = 120 and s = 16.33 (scikit-learn uses the population standard deviation: the square root of (400 + 0 + 400) / 3).
  2. The training values become about -1.225, 0 and 1.225: mean 0, standard deviation 1.
  3. scaler.mean_ prints [120.] and scaler.scale_ prints [16.33].
  4. The test value 130 is transformed with the training statistics: (130 - 120) / 16.33 = about 0.612, printed as [[0.612]].

If you wrongly called fit_transform on the one-row test set, its mean would become 130 and the value would turn into 0, which no longer means the same thing as in training.

Common mistakes

  • Calling fit_transform on the test set. The test set must be transformed with the training statistics only.
  • Scaling before splitting. Statistics computed on all rows include the test rows: a form of data leakage.
  • Expecting the scaled test set to have mean exactly 0. Only the training set is centred exactly.
  • Looking for column names after scaling. The output is a NumPy array; keep a copy of the column names if you need them.

Summary

  • Features with very different ranges should be standardised so the model treats them fairly.
  • StandardScaler computes z = (x - u) / s for every column.
  • fit learns the mean and standard deviation; transform applies them; fit_transform does both.
  • Use fit_transform on x_train and only transform on x_test to avoid leakage.
  • Result: NumPy arrays of shape (614, 8) and (154, 8) on comparable scales.

Flashcards · 3.P4 Standardization with StandardScaler

8 words
WordMeaningExample
  • Standardisation (z-score)

    Rescaling a value as z = (x - u) / s so the column has mean 0 and standard deviation 1.

    Glucose 140 with u = 120, s = 16.33 -> 1.225

  • StandardScaler

    scikit-learn transformer that standardises each column using its mean and standard deviation.

    scaler = StandardScaler()

  • fit()

    Learns the statistics (mean and standard deviation of each column) without changing the data.

    scaler.fit(x_train)

  • transform()

    Applies already-learned statistics to data; used alone on the test set.

    x_test = scaler.transform(x_test)

  • fit_transform()

    Fits on the data and transforms it in one call; used on the training set.

    x_train = scaler.fit_transform(x_train)

  • Data leakage (scaling)

    Letting test-set statistics influence preprocessing, which makes evaluation look better than reality.

    Calling fit_transform on x_test.

  • scaler.mean_ / scaler.scale_

    Attributes holding the per-column mean and standard deviation learned by fit.

    scaler.mean_ -> [120.]

  • Feature range imbalance

    Features measured on very different scales, which can make a model over-weight large-valued features.

    Pregnancies 0-17 vs Glucose up to about 200.