Period 7 · Handling Missing Values
[video] Lecture video — Period 7: Handling Missing Values — /api/lesson-media/videos/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794847907374-0b601d5ceda5c034.mp4
[markdown] content available only on the platform
[markdown] content available only on the platform
Flashcards · Handling Missing Values
Missing marker — A text value such as blank, `?` or `N/A` used in a file to mean the value is unknown.
`" N/A "` must be stripped before it can be recognized.
pd.NA — The pandas missing-value marker used by nullable dtypes such as `string` and `Float64`.
`q.replace({"?": pd.NA, "N/A": pd.NA})`
Float64 (nullable) — A pandas numeric dtype that can store decimal numbers and missing values together.
`df["quantity"] = q.astype("Float64")`
str.strip() — Removes spaces and other whitespace from both ends of each text value.
`" N/A "` becomes `"N/A"`.
count() vs len() — `count()` gives the number of non-missing values; `len()` gives the number of rows.
count 4, len 6
dropna(subset=...) — Removes rows with missing values only in the listed columns.
`df.dropna(subset=["quantity"])`
Median imputation — Filling missing values with the median of the observed values; it reduces variation.
`df["quantity"].fillna(df["quantity"].median())`
Imputation flag — A Boolean column, created before filling, that marks which values were estimated.
`df.assign(quantity_imputed=df["quantity"].isna())`
Keep and flag — Keeping rows with missing values and marking them, so other fields stay usable without guessing.
Mean 5 reported as based on 4 observed transactions.
[pdf] Lecture slides — Period 7 (PDF) — /api/lesson-media/pdfs/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794873715498-5270b1991d5d34b4.pdf
[markdown] content available only on the platform
