You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
Real data almost always has empty cells. If you ignore them, the model cannot run; if you handle them incorrectly, the model learns fabricated information.
Not all empty cells are the same:
| Type of missingness | What it means | Example |
|---|---|---|
| Missing completely at random | The empty cell is unrelated to the data | A sensor loses power for a few minutes |
| Missing at random (conditional) | The probability of missingness depends on another column | High earners hesitate to report their income |
| Missing not at random | The probability of missingness depends on the missing value itself | Seriously ill patients drop out of a survey |
The last two are more dangerous: the missingness itself carries information. Filling it in carelessly erases that signal.
1. Drop rows — use only when the number of rows with missing values is very small (below ~5%) and the missingness is random.
data = data.dropna() # drop every row with any empty cell
data = data.dropna(subset=["BMI"]) # only consider the BMI column
2. Fill with a statistic (imputation) — keep the rows and replace empty cells with a representative value.
data["BMI"] = data["BMI"].fillna(data["BMI"].median()) # numerical: median is safer than mean
data["city"] = data["city"].fillna(data["city"].mode()[0]) # categorical: the most frequent value
Why the median is usually better than the mean: the mean is pulled off by outliers. A single 100 billion VND villa in a housing-price dataset will pull the mean up, so every empty cell gets filled with an unrealistic value. The median is not affected this way.
3. Add an indicator column — when the missingness itself is meaningful:
data["BMI_missing"] = data["BMI"].isna().astype(int)
data["BMI"] = data["BMI"].fillna(data["BMI"].median())
This keeps both: the filled value so the model can run, and the signal "this spot was originally empty".
Computing the median over the whole dataset before splitting into train/test. The median has then already "seen" the test set — data leakage. You must split first, compute the median on the training set, and then apply that same value to the test set.
data.isna().sum() # count empty cells per column
data.isna().mean().sort_values() # missing rate, sorted ascending
By the end of this lesson you can:
Key idea: Empty cells must be handled before training, but how you fill them matters. Compute any fill value from the training set only, prefer the median for numbers, and keep an indicator column when the fact that a value is missing carries information.
Real data almost always has empty cells. Ignore them and most models will not run; handle them carelessly and the model learns invented information. Not all gaps are the same:
| Type | Meaning | Example |
|---|---|---|
| Missing completely at random | The gap has nothing to do with the data | A sensor loses power for a few minutes |
| Missing at random (conditional) | The chance of a gap depends on another column | High earners are more reluctant to report income |
| Missing not at random | The chance of a gap depends on the missing value itself | Seriously ill patients drop out of a survey |
The last two are more dangerous because the missingness itself carries information. Filling the gaps blindly erases that signal.
1. Drop rows. Use only when very few rows are affected (below about 5%) and the gaps are random. data.dropna() removes every row with any gap; data.dropna(subset=["BMI"]) looks only at the BMI column.
2. Impute a statistic. Keep the rows and fill gaps with a representative value:
| Column type | Fill with | pandas |
|---|---|---|
| Numerical | Median (safer than the mean) | data["BMI"].fillna(median_value) |
| Categorical | Most frequent value (mode) | data["city"].fillna(data["city"].mode()[0]) |
The mean is pulled by outliers. One 100 billion VND villa in a housing dataset raises the mean, so every gap would be filled with an unrealistic price. The median is not affected in this way.
3. Add an indicator column. When the gap is meaningful, record it before filling: data["BMI_missing"] = data["BMI"].isna().astype(int). The model then gets both a usable value and the signal "this cell was originally empty".
data.isna().sum() counts empty cells per column, and data.isna().mean().sort_values() shows the missing rate of each column, smallest first. Look at these numbers before deciding on a strategy.
The most common mistake is computing the median on the whole dataset before splitting into train and test. The median has then "seen" the test set, which is data leakage: your test score becomes slightly too optimistic. The correct order is: split first, compute the median (or mode) on the training set, then apply that same value to both train and test.
A small dataset has gaps in BMI and city. For a reproducible example we take the first six rows as train and the last two as test; in a real project you would use train_test_split.
import numpy as np
import pandas as pd
data = pd.DataFrame({
"BMI": [22.5, np.nan, 31.0, 27.4, np.nan, 24.1, np.nan, 45.0],
"city": ["Hanoi", "Hue", np.nan, "Hanoi", "Hanoi", np.nan, "Hue", np.nan],
})
print(data.isna().sum().tolist())
train = data.iloc[:6].copy()
test = data.iloc[6:].copy()
for df in (train, test):
df["BMI_missing"] = df["BMI"].isna().astype(int) # keep the signal
bmi_median = train["BMI"].median() # learned from train only
city_mode = train["city"].mode()[0]
print(bmi_median, data["BMI"].median())
for df in (train, test):
df["BMI"] = df["BMI"].fillna(bmi_median)
df["city"] = df["city"].fillna(city_mode)
print(test)
Expected output:
[3, 3]
25.75 27.4
BMI city BMI_missing
6 25.75 Hue 1
7 45.00 Hanoi 0
The training median is 25.75. Computed on all rows it would be 27.4, because the test value 45.0 leaks in. The test row with a missing BMI is filled with the training median and flagged with BMI_missing = 1; the missing city becomes the training mode, Hanoi.
_missing column before imputing.Flashcards · Handling Missing Values
9 wordsGaps that have no relationship with any data, so dropping or filling them introduces little bias.
Gaps whose probability depends on another observed column.
Gaps whose probability depends on the missing value itself, so the gap carries information.
pandas method that removes rows containing missing values, optionally only for chosen columns.
Filling numerical gaps with the column median, which is not pulled by outliers.
Filling categorical gaps with the most frequent value of the column.
A 0/1 column recording where a value was originally missing, kept alongside the filled value.
Information from the test set influencing training or preprocessing, making scores look better than reality.
Gives the share of missing values in each column.