You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
After this lesson you will be able to: use SimpleImputer to fill in empty cells, combine it with a scaler in a Pipeline, and avoid data leakage by following the rule "fit on train only".
import numpy as np
from sklearn.impute import SimpleImputer
x_train = np.array([[1.0], [2.0], [np.nan], [9.0]])
for strategy in ["mean", "median"]:
imp = SimpleImputer(strategy=strategy).fit(x_train)
print(strategy, imp.statistics_, imp.transform(x_train).ravel())
# mean [4.] [1. 2. 4. 9.]
# median [2.] [1. 2. 2. 9.]
statistics_ holds the values the imputer has "learned": mean = (1 + 2 + 9) / 3 = 4, median = 2 (the value 9 pulls the mean up, but not the median). For categorical columns, use most_frequent:
cats = np.array([["IT"], ["IT"], [np.nan], ["Econ"]], dtype=object)
imp = SimpleImputer(strategy="most_frequent").fit(cats)
print(imp.statistics_, imp.transform(cats).ravel()) # ['IT'] ['IT' 'IT' 'IT' 'Econ']
x_test = np.array([[np.nan], [20.0]])
imp = SimpleImputer(strategy="mean").fit(x_train) # learn from train ONLY
print(imp.transform(x_test).ravel()) # [ 4. 20.]
leaky = SimpleImputer(strategy="mean").fit(np.vstack([x_train, x_test]))
print(leaky.statistics_) # [8.] = (1+2+9+20)/4
wrong = SimpleImputer(strategy="mean").fit_transform(x_test)
print(wrong.ravel()) # [20. 20.]
The training mean is 4, but computed over train and test together it becomes 8: the model has "peeked" at the test value 20. Calling fit_transform on the test set is even worse, because the empty cell is filled using the test data (20), which you will not have at deployment time. The rule: fit/fit_transform only on train; test and new data only get transform.
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipe = Pipeline([
("imputer", SimpleImputer(strategy="mean")),
("scaler", StandardScaler()),
])
pipe.fit(x_train)
print(pipe["imputer"].statistics_, pipe["scaler"].mean_) # [4.] [4.]
print(pipe["scaler"].scale_.round(3)) # [3.082]
print(pipe.transform(x_test).ravel().round(3)) # [0. 5.191]
pipe.fit runs the imputer and then the scaler, both on train. Mean imputation does not change the mean, so the two 4 values coincide. On the test set, the empty cell is filled with 4 and then standardized to 0; the value 20 becomes (20 − 4) / 3.082 = 5.191.
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
train = pd.DataFrame({"age": [20, 22, np.nan, 26], "major": ["IT", "IT", np.nan, "Econ"]})
test = pd.DataFrame({"age": [24, np.nan], "major": ["Econ", "IT"]})
prep = ColumnTransformer([
("num", Pipeline([("imp", SimpleImputer(strategy="median")),
("sc", StandardScaler())]), ["age"]),
("cat", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
("oh", OneHotEncoder())]), ["major"]),
])
print(prep.fit_transform(train).shape) # (4, 3)
print(prep.transform(test).shape) # (2, 3)
print(prep.get_feature_names_out()) # ['num__age' 'cat__major_Econ' 'cat__major_IT']
Each group of columns has its own pipeline: 1 numerical column + 2 one-hot columns (Econ, IT) = 3 columns.
fit_transform (or fit) on the test set."?" or "N/A": the imputer only recognizes NaN, so convert them to NaN first.statistics_ shows the values used for filling; the mean is sensitive to outliers, the median is not.fit only on train; the test set only gets transform.Pipeline and ColumnTransformer enforce the rule above automatically.Flashcards · 1.11 Lab: Handling Missing Values and Pipelines
10 wordsscikit-learn class in sklearn.impute that fills missing values using a chosen strategy.
Imputes with the column median, which is not pulled upwards or downwards by outliers.
Imputes with the most common value; works for categorical columns.
Fitted imputer attribute holding the fill value learned for each column.
Learn imputation and scaling statistics from training data; test and new data only get transform.
Refitting on the test set fills gaps with test data you will not have at deployment time.
Chains imputation and scaling so both are fitted on train in order and applied consistently.
Applies separate pipelines to numerical and categorical columns of one table.
Lists the output column names produced by a fitted transformer.
Markers like "?" or "N/A" are not recognised as missing; convert them to NaN first.