You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
This final period reviews the full workflow. Start with the original data and keep an unchanged copy. Explore the table to find missing values, repeated records, unusual values, and broken business rules. Then choose suitable cleaning and transformation steps. Finally, rerun the notebook and check the outputs. A useful report explains both the findings and the limits of the data. Each decision needs a clear reason and a check of its effect.
By the end of this lesson you can:
Key idea: Data cleaning requires judgment. A method is useful when it fits the business question and its effect is clear, checked and reported.
Each decision needs a reason and a check of its effect. This is the same sequence you met in Period 1, now with the tools from Periods 2 to 14.
| Option | When it is reasonable | What to record |
|---|---|---|
| Keep and flag | The rest of the row still supports the analysis | Which fields are missing and how results use the row |
| Remove | An essential field is unavailable | How many rows, and which groups lose records |
| Fill | A justified rule or group estimate exists | The rule, a flag for filled values, and the change in totals |
Filling changes the data, so record it. A missing discount does not automatically mean that the customer received no discount.
First define what makes a record unique. Two purchases by the same customer can both be valid, so customer_id alone is not a duplicate test. Remove a record only when evidence confirms it is a duplicate.
For an invalid date or price, use a trusted source to correct it, or separate it for review. An IQR rule can flag a large sale, but the sale may be real. Any decision to cap or remove it needs a reason and a check of its impact.
The lecture's final example shows how one assumption changes a total. Three sales are each worth 100 before discount. Their discounts are 0%, missing and 20%.
Step by step:
import pandas as pd
sales = pd.DataFrame({"gross": [100, 100, 100],
"discount_rate": [0.0, None, 0.2]})
sales["revenue"] = sales["gross"] * (1 - sales["discount_rate"])
known = sales["revenue"].sum() # NaN rows are skipped
coverage = sales["revenue"].notna().sum()
assumed = (sales["gross"] * (1 - sales["discount_rate"].fillna(0))).sum()
print(known, f"{coverage} of {len(sales)} sales", assumed, assumed - known)
Expected output: 180.0 2 of 3 sales 280.0 100.0. Notice that sum() silently skips the missing revenue, which is exactly why you must report coverage next to the total.
Keep practising with new datasets, and always be able to explain the choices you made.
Flashcards · Review: The Full Workflow
8 wordsKeep the raw copy, explore, clean and transform with reasons, rerun and check, then report findings and limits.
A total calculated only from records with complete information, reported with its coverage.
The share of records that a result actually includes.
The change in a result caused only by an assumed value, which should be stated.
Testing whether a different reasonable treatment changes the conclusion.
A record kept visible in a review table or log because evidence to fix it is missing.
Comparing row counts, missing rates and key totals around each important treatment.
pandas `sum()` ignores missing values by default, so a total can be silently partial.