Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1: Data Preparation and Model Foundations

Capstone Practice: Planning Preprocessing for a Real Dataset

Print / Save as PDF

Capstone Practice: Planning Preprocessing

This lesson brings together the knowledge of Chapter 1 — data types, feature types, data quality, normalization, and encoding — into a preprocessing plan for a real dataset. Do it after you have completed units 1.1 through 1.10.

The task is in the Assignment section below.

Study notes

By the end of this lesson you can:

  • Inspect a new dataset and classify every column by data type and feature type.
  • List the data quality problems to check before modelling.
  • Choose a treatment for each column: imputation, scaling and encoding.
  • Turn your plan into a leak-free scikit-learn preprocessing object fitted on training data only.

Key idea: A good preprocessing plan is a column-by-column decision table built from Chapter 1: identify the target and each feature's type, check quality problems, choose imputation, scaling and encoding, and fit everything after splitting, on the training set only.

Step 1: understand the data

Start with the questions from lessons 1.2 to 1.6:

  • What type of data is it? This course works with tabular data.
  • What is the target? Is it a number (regression) or a label (classification)?
  • For each feature: is it numerical (discrete or continuous) or categorical (nominal, ordinal or boolean)? Watch for codes stored as digits that are really categories.
  • Is the data labeled at all? Supervised learning needs a known target for every training row.

Step 2: check data quality

Use the checklist from lesson 1.7 and the readings:

ProblemHow to checkTypical action
Missing valuesdf.isna().sum(), df.isna().mean()Drop if very few and random; otherwise impute; add an indicator if missingness is meaningful
Duplicatesdf.duplicated().sum()drop_duplicates()
Wrong formatsLook at unique values of text columns, dates and pricesConvert to one consistent format
OutliersSummary statistics, scatter plotsInvestigate; remove only clear errors; prefer the median and z-score when outliers stay
Low variancedf.nunique()Consider dropping columns that are almost constant

Step 3: choose a treatment for each column

Write one row per column in a plan table: column name, type, imputation, scaling or encoding, and the reason. The rules from lessons 1.8 and 1.9 and the readings apply directly:

  • Numerical columns: impute with the median, then scale (z-score if there are outliers, Min-Max if the bounds are known and the data is clean).
  • Ordinal columns: impute with the most frequent value, then ordinal encoding with the order written out explicitly.
  • Nominal and boolean columns: impute with the most frequent value, then one-hot encoding.
  • The target column is never scaled or encoded together with the features.

Step 4: split first, then fit

Every statistic in the plan (medians, modes, means, standard deviations, category lists) must be learned from the training set only. Split the data first, use stratification if the target classes are imbalanced, then fit the preprocessing on train and only transform validation and test data. Putting the whole plan inside a ColumnTransformer and Pipeline enforces this automatically.

Worked example

Plan and implement preprocessing for a housing table with square_feet, num_bedrooms, tier, city, has_garden and the target price.

ColumnTypeImputationTransformation
square_feetNumerical, continuousMedianStandardScaler
num_bedroomsNumerical, discreteMedianStandardScaler
tierCategorical, ordinalMost frequentOrdinalEncoder (Affordable < Mid-range < Luxury)
cityCategorical, nominalMost frequentOneHotEncoder
has_gardenCategorical, booleanMost frequentOneHotEncoder
priceTarget (regression)Not imputedNot transformed
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

houses = pd.DataFrame({
    "square_feet":  [60, np.nan, 80, 65, 45, 70, 90, 55],
    "num_bedrooms": [3, 2, 4, 3, 2, np.nan, 4, 2],
    "tier": ["Mid-range", "Affordable", "Luxury", np.nan, "Affordable", "Mid-range", "Luxury", "Affordable"],
    "city": ["Hanoi", "Hue", "Ho Chi Minh City", "Hanoi", np.nan, "Hue", "Hanoi", "Ho Chi Minh City"],
    "has_garden": ["No", "Yes", "Yes", "No", "No", "Yes", np.nan, "No"],
    "price": [3.2, 2.1, 5.0, 2.9, 1.8, 3.0, 5.5, 2.4],
})
train, test = houses.iloc[:6], houses.iloc[6:]   # in practice: train_test_split
X_train, y_train = train.drop(columns="price"), train["price"]
X_test, y_test = test.drop(columns="price"), test["price"]

prep = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                      ("sc", StandardScaler())]), ["square_feet", "num_bedrooms"]),
    ("ord", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
                      ("enc", OrdinalEncoder(categories=[["Affordable", "Mid-range", "Luxury"]]))]), ["tier"]),
    ("nom", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
                      ("enc", OneHotEncoder(handle_unknown="ignore"))]), ["city", "has_garden"]),
])
X_train_ready = prep.fit_transform(X_train)   # learn statistics from train only
X_test_ready = prep.transform(X_test)          # reuse them on test
print(X_train_ready.shape, X_test_ready.shape)
print(prep.get_feature_names_out())

Expected output: the shapes (6, 8) (2, 8), then eight column names: num__square_feet, num__num_bedrooms, ord__tier, three nom__city_... columns (Hanoi, Ho Chi Minh City, Hue) and two nom__has_garden_... columns (No, Yes). Two numerical columns, one ordinal column and five one-hot columns give 8 features. The missing has_garden in the test set is filled with the training mode, not with test data.

Common mistakes

  • Preprocessing before splitting. Medians, scalers and encoders fitted on all rows leak test information.
  • Treating every text column the same way. Decide ordinal or nominal per column; the encoding depends on it.
  • Scaling or encoding the target with the features. Keep price (or any target) separate.
  • Planning without looking. Check missing rates, duplicates, formats and outliers before choosing treatments.

Summary

  • Start by identifying the data type, the target and every feature's type.
  • Check missing values, duplicates, formats, outliers and low-variance columns.
  • Write a plan table: one row per column with imputation and transformation.
  • Split first, then fit all preprocessing on the training set only.
  • ColumnTransformer inside a Pipeline turns the plan into leak-free code.

Flashcards · Capstone Practice: Planning Preprocessing for a Real Dataset

8 words
WordMeaningExample
  • Preprocessing plan

    A column-by-column table stating each feature's type, how to impute it and how to scale or encode it.

    tier: ordinal, most frequent, OrdinalEncoder.

  • Data audit

    Checking a dataset for missing values, duplicates, wrong formats, outliers and low variance before modelling.

    df.isna().sum(); df.duplicated().sum()

  • df.nunique()

    Counts distinct values per column; helps spot near-constant columns and categorical features.

    houses.nunique()

  • SimpleImputer

    scikit-learn transformer that fills missing values with the mean, median, most frequent value or a constant.

    SimpleImputer(strategy="median")

  • Per-column treatment

    Choosing imputation and transformation separately for each feature according to its type.

    Median plus StandardScaler for square_feet.

  • Fit on train only

    All preprocessing statistics are learned from training rows and merely applied to other data.

    prep.fit_transform(X_train); prep.transform(X_test)

  • ColumnTransformer

    Runs a different preprocessing pipeline on each group of columns and joins the outputs.

    ("nom", OneHotEncoder(), ["city", "has_garden"])

  • Target separation

    Keeping the target column out of the feature preprocessing so it is never scaled or encoded with inputs.

    X = df.drop(columns="price"); y = df["price"]