Handling Missing Values
[markdown] content available only on the platform
[markdown] content available only on the platform
Flashcards · Handling Missing Values
Missing completely at random — Gaps that have no relationship with any data, so dropping or filling them introduces little bias.
A sensor briefly loses power.
Missing at random — Gaps whose probability depends on another observed column.
High earners skip the income question more often.
Missing not at random — Gaps whose probability depends on the missing value itself, so the gap carries information.
Seriously ill patients drop out of a survey.
df.dropna() — pandas method that removes rows containing missing values, optionally only for chosen columns.
data.dropna(subset=["BMI"])
Median imputation — Filling numerical gaps with the column median, which is not pulled by outliers.
data["BMI"].fillna(train["BMI"].median())
Mode imputation — Filling categorical gaps with the most frequent value of the column.
data["city"].fillna(data["city"].mode()[0])
Indicator column — A 0/1 column recording where a value was originally missing, kept alongside the filled value.
data["BMI_missing"] = data["BMI"].isna().astype(int)
Data leakage — Information from the test set influencing training or preprocessing, making scores look better than reality.
Computing the median on all rows before splitting.
df.isna().mean() — Gives the share of missing values in each column.
data.isna().mean().sort_values()
