Period 8 · Duplicates and Inconsistent Labels
[video] Lecture video — Period 8: Duplicates and Inconsistent Labels — /api/lesson-media/videos/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794847908839-4d31dc39f76f95fd.mp4
[markdown] content available only on the platform
[markdown] content available only on the platform
Flashcards · Duplicates and Inconsistent Labels
Exact duplicate — A row whose values are identical to an earlier row in every compared column.
Rows A and B: T01, North, 2.
Key conflict — Two or more rows share the business key but disagree on other values.
T02 recorded with quantity 3 and quantity 5.
drop_duplicates() — Removes rows marked as duplicates, keeping the first occurrence by default.
`work.drop_duplicates(subset=business)`
str.casefold() — Converts text to a consistent case for comparison, more thoroughly than `lower()`.
`"NORTH"` becomes `"north"`.
Regular expression \s+ — A pattern matching one or more whitespace characters, used to collapse inner spaces.
`str.replace(r"\s+", " ", regex=True)`
Mapping dictionary — An approved lookup that turns each known label form into one standard label.
`{"north": "North", "n.": "North"}`
Series.map() — Replaces each value using a dictionary; values not in the dictionary become missing.
`region.map(REGION_MAP)`
Unknown mask — Flags labels that were present before mapping but have no approved match after mapping.
`region.notna() & work["region_clean"].isna()`
Review table — A separate table that keeps rows you cannot resolve without evidence, instead of deleting them.
Both T02 rows wait for a verified source record.
[pdf] Lecture slides — Period 8 (PDF) — /api/lesson-media/pdfs/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794873718237-173d2cc4632169d1.pdf
