Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1: Data Preparation and Model Foundations

Handling Missing Values

Print / Save as PDF

Handling Missing Values

Real data almost always has empty cells. If you ignore them, the model cannot run; if you handle them incorrectly, the model learns fabricated information.

Why data goes missing — and why it matters

Not all empty cells are the same:

Type of missingnessWhat it meansExample
Missing completely at randomThe empty cell is unrelated to the dataA sensor loses power for a few minutes
Missing at random (conditional)The probability of missingness depends on another columnHigh earners hesitate to report their income
Missing not at randomThe probability of missingness depends on the missing value itselfSeriously ill patients drop out of a survey

The last two are more dangerous: the missingness itself carries information. Filling it in carelessly erases that signal.

Three common ways to handle it

1. Drop rows — use only when the number of rows with missing values is very small (below ~5%) and the missingness is random.

data = data.dropna()              # drop every row with any empty cell
data = data.dropna(subset=["BMI"])  # only consider the BMI column

2. Fill with a statistic (imputation) — keep the rows and replace empty cells with a representative value.

data["BMI"] = data["BMI"].fillna(data["BMI"].median())   # numerical: median is safer than mean
data["city"] = data["city"].fillna(data["city"].mode()[0])  # categorical: the most frequent value

Why the median is usually better than the mean: the mean is pulled off by outliers. A single 100 billion VND villa in a housing-price dataset will pull the mean up, so every empty cell gets filled with an unrealistic value. The median is not affected this way.

3. Add an indicator column — when the missingness itself is meaningful:

data["BMI_missing"] = data["BMI"].isna().astype(int)
data["BMI"] = data["BMI"].fillna(data["BMI"].median())

This keeps both: the filled value so the model can run, and the signal "this spot was originally empty".

The most common mistake

Computing the median over the whole dataset before splitting into train/test. The median has then already "seen" the test set — data leakage. You must split first, compute the median on the training set, and then apply that same value to the test set.

Quick check

data.isna().sum()                       # count empty cells per column
data.isna().mean().sort_values()        # missing rate, sorted ascending

Study notes

By the end of this lesson you can:

  • Distinguish three kinds of missingness and explain why two of them are risky.
  • Choose between dropping rows, imputing a statistic and adding an indicator column.
  • Explain why the median is usually safer than the mean for numerical imputation.
  • Impute without data leakage by computing statistics on the training set only.

Key idea: Empty cells must be handled before training, but how you fill them matters. Compute any fill value from the training set only, prefer the median for numbers, and keep an indicator column when the fact that a value is missing carries information.

Why data goes missing

Real data almost always has empty cells. Ignore them and most models will not run; handle them carelessly and the model learns invented information. Not all gaps are the same:

TypeMeaningExample
Missing completely at randomThe gap has nothing to do with the dataA sensor loses power for a few minutes
Missing at random (conditional)The chance of a gap depends on another columnHigh earners are more reluctant to report income
Missing not at randomThe chance of a gap depends on the missing value itselfSeriously ill patients drop out of a survey

The last two are more dangerous because the missingness itself carries information. Filling the gaps blindly erases that signal.

Three common ways to handle gaps

1. Drop rows. Use only when very few rows are affected (below about 5%) and the gaps are random. data.dropna() removes every row with any gap; data.dropna(subset=["BMI"]) looks only at the BMI column.

2. Impute a statistic. Keep the rows and fill gaps with a representative value:

Column typeFill withpandas
NumericalMedian (safer than the mean)data["BMI"].fillna(median_value)
CategoricalMost frequent value (mode)data["city"].fillna(data["city"].mode()[0])

The mean is pulled by outliers. One 100 billion VND villa in a housing dataset raises the mean, so every gap would be filled with an unrealistic price. The median is not affected in this way.

3. Add an indicator column. When the gap is meaningful, record it before filling: data["BMI_missing"] = data["BMI"].isna().astype(int). The model then gets both a usable value and the signal "this cell was originally empty".

Checking how much is missing

data.isna().sum() counts empty cells per column, and data.isna().mean().sort_values() shows the missing rate of each column, smallest first. Look at these numbers before deciding on a strategy.

The data leakage trap

The most common mistake is computing the median on the whole dataset before splitting into train and test. The median has then "seen" the test set, which is data leakage: your test score becomes slightly too optimistic. The correct order is: split first, compute the median (or mode) on the training set, then apply that same value to both train and test.

Worked example

A small dataset has gaps in BMI and city. For a reproducible example we take the first six rows as train and the last two as test; in a real project you would use train_test_split.

import numpy as np
import pandas as pd

data = pd.DataFrame({
    "BMI":  [22.5, np.nan, 31.0, 27.4, np.nan, 24.1, np.nan, 45.0],
    "city": ["Hanoi", "Hue", np.nan, "Hanoi", "Hanoi", np.nan, "Hue", np.nan],
})
print(data.isna().sum().tolist())

train = data.iloc[:6].copy()
test = data.iloc[6:].copy()

for df in (train, test):
    df["BMI_missing"] = df["BMI"].isna().astype(int)   # keep the signal

bmi_median = train["BMI"].median()       # learned from train only
city_mode = train["city"].mode()[0]
print(bmi_median, data["BMI"].median())

for df in (train, test):
    df["BMI"] = df["BMI"].fillna(bmi_median)
    df["city"] = df["city"].fillna(city_mode)
print(test)

Expected output:

[3, 3]
25.75 27.4
     BMI   city  BMI_missing
6  25.75    Hue            1
7  45.00  Hanoi            0

The training median is 25.75. Computed on all rows it would be 27.4, because the test value 45.0 leaks in. The test row with a missing BMI is filled with the training median and flagged with BMI_missing = 1; the missing city becomes the training mode, Hanoi.

Common mistakes

  • Computing fill values before splitting. Always split first, then learn medians and modes from train only.
  • Filling with the mean when outliers exist. Extreme values drag the mean; use the median.
  • Dropping many rows. If more than a small share of rows has gaps, dropping loses data and can bias the sample.
  • Filling gaps that carry meaning without an indicator. When high earners skip the income question, add a _missing column before imputing.

Summary

  • Gaps can be completely random, random given other columns, or related to the missing value itself.
  • Drop rows only when few are affected and the gaps are random.
  • Impute numbers with the median and categories with the most frequent value.
  • Add an indicator column when missingness is informative.
  • Split first and compute fill values on the training set only.

Flashcards · Handling Missing Values

9 words
WordMeaningExample
  • Missing completely at random

    Gaps that have no relationship with any data, so dropping or filling them introduces little bias.

    A sensor briefly loses power.

  • Missing at random

    Gaps whose probability depends on another observed column.

    High earners skip the income question more often.

  • Missing not at random

    Gaps whose probability depends on the missing value itself, so the gap carries information.

    Seriously ill patients drop out of a survey.

  • df.dropna()

    pandas method that removes rows containing missing values, optionally only for chosen columns.

    data.dropna(subset=["BMI"])

  • Median imputation

    Filling numerical gaps with the column median, which is not pulled by outliers.

    data["BMI"].fillna(train["BMI"].median())

  • Mode imputation

    Filling categorical gaps with the most frequent value of the column.

    data["city"].fillna(data["city"].mode()[0])

  • Indicator column

    A 0/1 column recording where a value was originally missing, kept alongside the filled value.

    data["BMI_missing"] = data["BMI"].isna().astype(int)

  • Data leakage

    Information from the test set influencing training or preprocessing, making scores look better than reality.

    Computing the median on all rows before splitting.

  • df.isna().mean()

    Gives the share of missing values in each column.

    data.isna().mean().sort_values()