Back to the lessonPreparing the printout…

Data Cleaning and Preprocessing with Python · Module 1 — Explore and Understand the Data

Period 3 · Measuring Data Quality: Missing Values, Duplicates and Categories

[video] Lecture video — Period 3: Measuring Data Quality: Missing Values, Duplicates and Categories — /api/lesson-media/videos/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794892495946-816c0abd05eb7967.mp4

[markdown] content available only on the platform

[markdown] content available only on the platform

Flashcards · Measuring Data Quality: Missing Values, Duplicates and Categories

df.isna() — Returns True where a value is a real missing value (None, NaN, NA) and False elsewhere.

`df["region"].isna().sum()` counts missing regions.

Missing percentage — The share of rows with a missing value, using all input rows as the denominator.

1 missing of 6 rows = 16.67%

df.duplicated() — Marks rows that repeat an earlier row; by default the first occurrence is not marked.

`df.duplicated().sum()` gives 1 for two identical rows.

keep=False — Option of `duplicated()` that marks every row in a repeated group, including the first.

`df.duplicated(subset=["transaction_id"], keep=False)`

subset — Limits the duplicate comparison to selected columns, such as the business key.

`subset=["transaction_id"]`

nunique() — Counts distinct values; missing values are excluded unless `dropna=False`.

`North`, `South`, `north` and None give 3 by default.

value_counts() — Lists each distinct value with how often it appears.

`df["region"].value_counts(dropna=False)`

Business rule check — A condition derived from the business meaning of a column, used to flag invalid rows.

`(df["quantity"] <= 0) | ~df["discount_rate"].between(0, 1)`

Quality report — A table summarizing missing counts, percentages and distinct values per column before cleaning.

One row per column: `missing_count`, `missing_pct`, `n_distinct`.

[pdf] Lecture slides — Period 3 (PDF) — /api/lesson-media/pdfs/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794873700315-2271fc8249e5294a.pdf

Data Cleaning and Preprocessing with Python · Period 3 · Measuring Data Quality: Missing Values, Duplicates and Categories · printed on November 20, 2026