Period 13 · Building a Rerunnable Cleaning Workflow
[video] Lecture video — Period 13: Building a Rerunnable Cleaning Workflow — /api/lesson-media/videos/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794846243337-b725847b1eb48e00.mp4
[markdown] content available only on the platform
[markdown] content available only on the platform
Flashcards · Building a Rerunnable Cleaning Workflow
Rerunnable notebook — A notebook that starts from the raw data and gives the same results whenever input, rules and software are unchanged.
Restart kernel, Run All, compare outputs.
Hidden state — Variables left in memory from cells that were run earlier or later deleted, which make results hard to repeat.
A `df` defined in a cell you removed.
Cleaning function — A function with a narrow purpose that takes a DataFrame, works on a copy and returns a result.
`def convert_numeric(frame): out = frame.copy(); ...; return out`
assert — A statement that stops execution with AssertionError when a required condition is false.
`assert clean["transaction_id"].is_unique, "IDs must be unique"`
Processing log — A table recording each step and how many rows it removed, sent to review or accepted.
6 input = 1 duplicate + 2 review + 3 accepted
pd.testing.assert_frame_equal() — Raises an error if two DataFrames differ, used to confirm a rerun gives the same result.
`assert_frame_equal(key(clean), key(clean2))`
index=False — Option of `to_csv` that stops pandas writing the row index as an extra column.
`clean.to_csv("output/sales_clean.csv", index=False)`
Read-back check — Reloading an exported file to confirm it exists, has the right shape and keeps values correctly.
`pd.read_csv(path, dtype={"transaction_id": str})`
Required field — A column that must be filled for a record to be accepted, unlike optional fields.
`unit_price` is required; a missing price sends the row to review.
[pdf] Lecture slides — Period 13 (PDF) — /api/lesson-media/pdfs/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794873731926-2ce4f2efb60004cb.pdf
[markdown] content available only on the platform
