Period 3 · Measuring Data Quality: Missing Values, Duplicates and Categories
[video] Lecture video — Period 3: Measuring Data Quality: Missing Values, Duplicates and Categories — /api/lesson-media/videos/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794892495946-816c0abd05eb7967.mp4
[markdown] content available only on the platform
[markdown] content available only on the platform
Flashcards · Measuring Data Quality: Missing Values, Duplicates and Categories
df.isna() — Returns True where a value is a real missing value (None, NaN, NA) and False elsewhere.
`df["region"].isna().sum()` counts missing regions.
Missing percentage — The share of rows with a missing value, using all input rows as the denominator.
1 missing of 6 rows = 16.67%
df.duplicated() — Marks rows that repeat an earlier row; by default the first occurrence is not marked.
`df.duplicated().sum()` gives 1 for two identical rows.
keep=False — Option of `duplicated()` that marks every row in a repeated group, including the first.
`df.duplicated(subset=["transaction_id"], keep=False)`
subset — Limits the duplicate comparison to selected columns, such as the business key.
`subset=["transaction_id"]`
nunique() — Counts distinct values; missing values are excluded unless `dropna=False`.
`North`, `South`, `north` and None give 3 by default.
value_counts() — Lists each distinct value with how often it appears.
`df["region"].value_counts(dropna=False)`
Business rule check — A condition derived from the business meaning of a column, used to flag invalid rows.
`(df["quantity"] <= 0) | ~df["discount_rate"].between(0, 1)`
Quality report — A table summarizing missing counts, percentages and distinct values per column before cleaning.
One row per column: `missing_count`, `missing_pct`, `n_distinct`.
[pdf] Lecture slides — Period 3 (PDF) — /api/lesson-media/pdfs/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794873700315-2271fc8249e5294a.pdf
