You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
This lesson brings together the knowledge of Chapter 1 — data types, feature types, data quality, normalization, and encoding — into a preprocessing plan for a real dataset. Do it after you have completed units 1.1 through 1.10.
The task is in the Assignment section below.
By the end of this lesson you can:
Key idea: A good preprocessing plan is a column-by-column decision table built from Chapter 1: identify the target and each feature's type, check quality problems, choose imputation, scaling and encoding, and fit everything after splitting, on the training set only.
Start with the questions from lessons 1.2 to 1.6:
Use the checklist from lesson 1.7 and the readings:
| Problem | How to check | Typical action |
|---|---|---|
| Missing values | df.isna().sum(), df.isna().mean() | Drop if very few and random; otherwise impute; add an indicator if missingness is meaningful |
| Duplicates | df.duplicated().sum() | drop_duplicates() |
| Wrong formats | Look at unique values of text columns, dates and prices | Convert to one consistent format |
| Outliers | Summary statistics, scatter plots | Investigate; remove only clear errors; prefer the median and z-score when outliers stay |
| Low variance | df.nunique() | Consider dropping columns that are almost constant |
Write one row per column in a plan table: column name, type, imputation, scaling or encoding, and the reason. The rules from lessons 1.8 and 1.9 and the readings apply directly:
Every statistic in the plan (medians, modes, means, standard deviations, category lists) must be learned from the training set only. Split the data first, use stratification if the target classes are imbalanced, then fit the preprocessing on train and only transform validation and test data. Putting the whole plan inside a ColumnTransformer and Pipeline enforces this automatically.
Plan and implement preprocessing for a housing table with square_feet, num_bedrooms, tier, city, has_garden and the target price.
| Column | Type | Imputation | Transformation |
|---|---|---|---|
square_feet | Numerical, continuous | Median | StandardScaler |
num_bedrooms | Numerical, discrete | Median | StandardScaler |
tier | Categorical, ordinal | Most frequent | OrdinalEncoder (Affordable < Mid-range < Luxury) |
city | Categorical, nominal | Most frequent | OneHotEncoder |
has_garden | Categorical, boolean | Most frequent | OneHotEncoder |
price | Target (regression) | Not imputed | Not transformed |
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler
houses = pd.DataFrame({
"square_feet": [60, np.nan, 80, 65, 45, 70, 90, 55],
"num_bedrooms": [3, 2, 4, 3, 2, np.nan, 4, 2],
"tier": ["Mid-range", "Affordable", "Luxury", np.nan, "Affordable", "Mid-range", "Luxury", "Affordable"],
"city": ["Hanoi", "Hue", "Ho Chi Minh City", "Hanoi", np.nan, "Hue", "Hanoi", "Ho Chi Minh City"],
"has_garden": ["No", "Yes", "Yes", "No", "No", "Yes", np.nan, "No"],
"price": [3.2, 2.1, 5.0, 2.9, 1.8, 3.0, 5.5, 2.4],
})
train, test = houses.iloc[:6], houses.iloc[6:] # in practice: train_test_split
X_train, y_train = train.drop(columns="price"), train["price"]
X_test, y_test = test.drop(columns="price"), test["price"]
prep = ColumnTransformer([
("num", Pipeline([("imp", SimpleImputer(strategy="median")),
("sc", StandardScaler())]), ["square_feet", "num_bedrooms"]),
("ord", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
("enc", OrdinalEncoder(categories=[["Affordable", "Mid-range", "Luxury"]]))]), ["tier"]),
("nom", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
("enc", OneHotEncoder(handle_unknown="ignore"))]), ["city", "has_garden"]),
])
X_train_ready = prep.fit_transform(X_train) # learn statistics from train only
X_test_ready = prep.transform(X_test) # reuse them on test
print(X_train_ready.shape, X_test_ready.shape)
print(prep.get_feature_names_out())
Expected output: the shapes (6, 8) (2, 8), then eight column names: num__square_feet, num__num_bedrooms, ord__tier, three nom__city_... columns (Hanoi, Ho Chi Minh City, Hue) and two nom__has_garden_... columns (No, Yes). Two numerical columns, one ordinal column and five one-hot columns give 8 features. The missing has_garden in the test set is filled with the training mode, not with test data.
price (or any target) separate.ColumnTransformer inside a Pipeline turns the plan into leak-free code.Flashcards · Capstone Practice: Planning Preprocessing for a Real Dataset
8 wordsA column-by-column table stating each feature's type, how to impute it and how to scale or encode it.
Checking a dataset for missing values, duplicates, wrong formats, outliers and low variance before modelling.
Counts distinct values per column; helps spot near-constant columns and categorical features.
scikit-learn transformer that fills missing values with the mean, median, most frequent value or a constant.
Choosing imputation and transformation separately for each feature according to its type.
All preprocessing statistics are learned from training rows and merely applied to other data.
Runs a different preprocessing pipeline on each group of columns and joins the outputs.
Keeping the target column out of the feature preprocessing so it is never scaled or encoded with inputs.