Back to the lessonPreparing the printout…

Data Cleaning and Preprocessing with Python · Module 2 — Clean the Data

Period 8 · Duplicates and Inconsistent Labels

[video] Lecture video — Period 8: Duplicates and Inconsistent Labels — /api/lesson-media/videos/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794847908839-4d31dc39f76f95fd.mp4

[markdown] content available only on the platform

[markdown] content available only on the platform

Flashcards · Duplicates and Inconsistent Labels

Exact duplicate — A row whose values are identical to an earlier row in every compared column.

Rows A and B: T01, North, 2.

Key conflict — Two or more rows share the business key but disagree on other values.

T02 recorded with quantity 3 and quantity 5.

drop_duplicates() — Removes rows marked as duplicates, keeping the first occurrence by default.

`work.drop_duplicates(subset=business)`

str.casefold() — Converts text to a consistent case for comparison, more thoroughly than `lower()`.

`"NORTH"` becomes `"north"`.

Regular expression \s+ — A pattern matching one or more whitespace characters, used to collapse inner spaces.

`str.replace(r"\s+", " ", regex=True)`

Mapping dictionary — An approved lookup that turns each known label form into one standard label.

`{"north": "North", "n.": "North"}`

Series.map() — Replaces each value using a dictionary; values not in the dictionary become missing.

`region.map(REGION_MAP)`

Unknown mask — Flags labels that were present before mapping but have no approved match after mapping.

`region.notna() & work["region_clean"].isna()`

Review table — A separate table that keeps rows you cannot resolve without evidence, instead of deleting them.

Both T02 rows wait for a verified source record.

[pdf] Lecture slides — Period 8 (PDF) — /api/lesson-media/pdfs/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794873718237-173d2cc4632169d1.pdf

Data Cleaning and Preprocessing with Python · Period 8 · Duplicates and Inconsistent Labels · printed on November 20, 2026