You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
After this unit you will: standardize data correctly with StandardScaler without leaking test data.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
x_train = scaler.fit_transform(x_train) # fit ONLY on the training set
x_test = scaler.transform(x_test) # the test set is only transformed
| Formula | |
|---|---|
| Standardization (z-score) | z = (x − u) / s where u is the mean and s is the standard deviation |
fit_transform learns the mean/std from the training set and then applies them to that same set; transform only applies the parameters that were already learned.x_train and x_test become NumPy arrays with shapes (614, 8) and (154, 8).By the end of this lesson you can:
StandardScalerfit, transform and fit_transformKey idea: Learn the mean and standard deviation from the training set only (
fit_transformon train), then reuse exactly those numbers on the test set (transformon test).
The eight diabetes features have very different ranges. For example, Pregnancies goes from 0 to 17, while Glucose goes up to about 200. As discussed in the theory part, when features have different ranges, a model can pay more attention to the features with larger values. To make the model see all inputs fairly, we put them on the same scale.
scikit-learn's StandardScaler implements standardisation. Its documentation gives the standardised value of a sample x as:
z = (x - u) / s
where u is the mean of the column and s is its standard deviation. After this transformation, each column of the training data has mean 0 and standard deviation 1.
The two methods used in the notebook are fit_transform and transform. The difference:
| Method | What it does |
|---|---|
fit | calculates the mean and standard deviation of each column; no transformation yet |
transform | applies z = (x - u) / s using the mean and standard deviation already calculated |
fit_transform | fit then transform in one call |
So, for the Pregnancies column, fit computes its mean and standard deviation, and transform takes every value, subtracts that mean and divides by that standard deviation. The same is done for every column with its own mean and standard deviation.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
x_train = scaler.fit_transform(x_train) # fit ONLY on the training set
x_test = scaler.transform(x_test) # the test set is only transformed
fit_transform: compute the statistics, then transform.transform it with the mean and standard deviation learned from training.Why? The test set represents new, unseen data. If we also fitted on the test set, information from it would leak into preprocessing and the evaluation would look better than it really is.
After scaling, x_train and x_test become NumPy arrays (no column names) with shapes (614, 8) and (154, 8). The values look completely different from the originals but are now on comparable scales. Because the test set uses the training statistics, its column means are close to 0, but not exactly 0.
A tiny Glucose column shows exactly what is learned and reused.
import pandas as pd
from sklearn.preprocessing import StandardScaler
train = pd.DataFrame({"Glucose": [100, 120, 140]})
test = pd.DataFrame({"Glucose": [130]})
scaler = StandardScaler()
print(scaler.fit_transform(train).ravel().round(3))
print(scaler.mean_, scaler.scale_.round(2))
print(scaler.transform(test).round(3))
Expected output and reasoning:
fit learns u = 120 and s = 16.33 (scikit-learn uses the population standard deviation: the square root of (400 + 0 + 400) / 3).scaler.mean_ prints [120.] and scaler.scale_ prints [16.33].[[0.612]].If you wrongly called fit_transform on the one-row test set, its mean would become 130 and the value would turn into 0, which no longer means the same thing as in training.
fit_transform on the test set. The test set must be transformed with the training statistics only.StandardScaler computes z = (x - u) / s for every column.fit learns the mean and standard deviation; transform applies them; fit_transform does both.fit_transform on x_train and only transform on x_test to avoid leakage.Flashcards · 3.P4 Standardization with StandardScaler
8 wordsRescaling a value as z = (x - u) / s so the column has mean 0 and standard deviation 1.
scikit-learn transformer that standardises each column using its mean and standard deviation.
Learns the statistics (mean and standard deviation of each column) without changing the data.
Applies already-learned statistics to data; used alone on the test set.
Fits on the data and transforms it in one call; used on the training set.
Letting test-set statistics influence preprocessing, which makes evaluation look better than reality.
Attributes holding the per-column mean and standard deviation learned by fit.
Features measured on very different scales, which can make a model over-weight large-valued features.