Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1: Data Preparation and Model Foundations

1.11 Lab: Handling Missing Values and Pipelines

Print / Save as PDF

Lab: Handling Missing Values and Pipelines

After this lesson you will be able to: use SimpleImputer to fill in empty cells, combine it with a scaler in a Pipeline, and avoid data leakage by following the rule "fit on train only".

1. SimpleImputer with different strategies

import numpy as np
from sklearn.impute import SimpleImputer

x_train = np.array([[1.0], [2.0], [np.nan], [9.0]])

for strategy in ["mean", "median"]:
    imp = SimpleImputer(strategy=strategy).fit(x_train)
    print(strategy, imp.statistics_, imp.transform(x_train).ravel())
# mean [4.] [1. 2. 4. 9.]
# median [2.] [1. 2. 2. 9.]

statistics_ holds the values the imputer has "learned": mean = (1 + 2 + 9) / 3 = 4, median = 2 (the value 9 pulls the mean up, but not the median). For categorical columns, use most_frequent:

cats = np.array([["IT"], ["IT"], [np.nan], ["Econ"]], dtype=object)
imp = SimpleImputer(strategy="most_frequent").fit(cats)
print(imp.statistics_, imp.transform(cats).ravel())   # ['IT'] ['IT' 'IT' 'IT' 'Econ']

2. Data leakage: fit on train, only transform on test

x_test = np.array([[np.nan], [20.0]])

imp = SimpleImputer(strategy="mean").fit(x_train)          # learn from train ONLY
print(imp.transform(x_test).ravel())                       # [ 4. 20.]

leaky = SimpleImputer(strategy="mean").fit(np.vstack([x_train, x_test]))
print(leaky.statistics_)                                   # [8.]  = (1+2+9+20)/4

wrong = SimpleImputer(strategy="mean").fit_transform(x_test)
print(wrong.ravel())                                       # [20. 20.]

The training mean is 4, but computed over train and test together it becomes 8: the model has "peeked" at the test value 20. Calling fit_transform on the test set is even worse, because the empty cell is filled using the test data (20), which you will not have at deployment time. The rule: fit/fit_transform only on train; test and new data only get transform.

3. Pipeline: imputer + scaler

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="mean")),
    ("scaler", StandardScaler()),
])
pipe.fit(x_train)
print(pipe["imputer"].statistics_, pipe["scaler"].mean_)   # [4.] [4.]
print(pipe["scaler"].scale_.round(3))                      # [3.082]
print(pipe.transform(x_test).ravel().round(3))             # [0.    5.191]

pipe.fit runs the imputer and then the scaler, both on train. Mean imputation does not change the mean, so the two 4 values coincide. On the test set, the empty cell is filled with 4 and then standardized to 0; the value 20 becomes (20 − 4) / 3.082 = 5.191.

4. ColumnTransformer for a table with both numerical and categorical columns

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

train = pd.DataFrame({"age": [20, 22, np.nan, 26], "major": ["IT", "IT", np.nan, "Econ"]})
test = pd.DataFrame({"age": [24, np.nan], "major": ["Econ", "IT"]})

prep = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                      ("sc", StandardScaler())]), ["age"]),
    ("cat", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
                      ("oh", OneHotEncoder())]), ["major"]),
])
print(prep.fit_transform(train).shape)   # (4, 3)
print(prep.transform(test).shape)        # (2, 3)
print(prep.get_feature_names_out())      # ['num__age' 'cat__major_Econ' 'cat__major_IT']

Each group of columns has its own pipeline: 1 numerical column + 2 one-hot columns (Econ, IT) = 3 columns.

Common mistakes

  • Imputing the median over the whole table before splitting into train/test.
  • Calling fit_transform (or fit) on the test set.
  • Missing cells written as "?" or "N/A": the imputer only recognizes NaN, so convert them to NaN first.

Key takeaways

  • statistics_ shows the values used for filling; the mean is sensitive to outliers, the median is not.
  • fit only on train; the test set only gets transform.
  • Pipeline and ColumnTransformer enforce the rule above automatically.

Flashcards · 1.11 Lab: Handling Missing Values and Pipelines

10 words
WordMeaningExample
  • SimpleImputer

    scikit-learn class in sklearn.impute that fills missing values using a chosen strategy.

    SimpleImputer(strategy="mean").fit(x_train)

  • strategy="median"

    Imputes with the column median, which is not pulled upwards or downwards by outliers.

    Values 1, 2, 9: the median is 2, the mean is 4.

  • strategy="most_frequent"

    Imputes with the most common value; works for categorical columns.

    ["IT", "IT", nan, "Econ"] -> nan becomes "IT"

  • statistics_

    Fitted imputer attribute holding the fill value learned for each column.

    imp.statistics_ -> [4.]

  • Fit on train only

    Learn imputation and scaling statistics from training data; test and new data only get transform.

    imp.fit(x_train); imp.transform(x_test)

  • fit_transform on test (mistake)

    Refitting on the test set fills gaps with test data you will not have at deployment time.

    Test [nan, 20] becomes [20, 20] instead of [4, 20].

  • Pipeline (imputer + scaler)

    Chains imputation and scaling so both are fitted on train in order and applied consistently.

    Pipeline([("imputer", SimpleImputer()), ("scaler", StandardScaler())])

  • ColumnTransformer

    Applies separate pipelines to numerical and categorical columns of one table.

    ("cat", Pipeline([...OneHotEncoder()]), ["major"])

  • get_feature_names_out()

    Lists the output column names produced by a fitted transformer.

    ['num__age' 'cat__major_Econ' 'cat__major_IT']

  • Placeholder missing values

    Markers like "?" or "N/A" are not recognised as missing; convert them to NaN first.

    df.replace("?", np.nan)

ML_Lab_1.11_notebook.zip· 2,2 KB