Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Data Cleaning and Preprocessing with Python

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 3 — Prepare, Report and Reproduce

Period 15 · Review: The Full Workflow

Print / Save as PDF
Lecture video — Period 15: Review: The Full Workflow

A review of the whole workflow from original data to report, with checks before and after each important treatment; course wrap-up.

Lesson overview

This final period reviews the full workflow. Start with the original data and keep an unchanged copy. Explore the table to find missing values, repeated records, unusual values, and broken business rules. Then choose suitable cleaning and transformation steps. Finally, rerun the notebook and check the outputs. A useful report explains both the findings and the limits of the data. Each decision needs a clear reason and a check of its effect.

Study notes

By the end of this lesson you can:

  • Describe the full workflow from the original file to a checked report.
  • Choose between keeping, removing and filling missing values, and record the effect.
  • Decide how to handle repeated records, invalid values and statistical outliers using evidence.
  • Calculate how an assumption changes a total, and report coverage honestly.
  • Run the final checks on both the data and the workflow before using the results.

Key idea: Data cleaning requires judgment. A method is useful when it fits the business question and its effect is clear, checked and reported.

The full workflow

  1. Start with the original data and keep an unchanged copy.
  2. Explore the table to find missing values, repeated records, unusual values and broken business rules.
  3. Choose suitable cleaning and transformation steps, each with a clear reason.
  4. Rerun the notebook and check the outputs.
  5. Report both the findings and the limits of the data.

Each decision needs a reason and a check of its effect. This is the same sequence you met in Period 1, now with the tools from Periods 2 to 14.

Missing values

OptionWhen it is reasonableWhat to record
Keep and flagThe rest of the row still supports the analysisWhich fields are missing and how results use the row
RemoveAn essential field is unavailableHow many rows, and which groups lose records
FillA justified rule or group estimate existsThe rule, a flag for filled values, and the change in totals

Filling changes the data, so record it. A missing discount does not automatically mean that the customer received no discount.

Repeated and unusual values

First define what makes a record unique. Two purchases by the same customer can both be valid, so customer_id alone is not a duplicate test. Remove a record only when evidence confirms it is a duplicate.

For an invalid date or price, use a trusted source to correct it, or separate it for review. An IQR rule can flag a large sale, but the sale may be real. Any decision to cap or remove it needs a reason and a check of its impact.

Final checks before using results

  • Compare row counts, missing rates and key totals before and after each important treatment.
  • Explain every exclusion and keep unresolved records visible in the log.
  • Rerun every cell from the raw file to confirm that the process works again.
  • Ask whether each conclusion has evidence, and whether a different reasonable treatment would change it.
  • Keep the notebook, cleaned data, quality log and short report together.

Worked example

The lecture's final example shows how one assumption changes a total. Three sales are each worth 100 before discount. Their discounts are 0%, missing and 20%.

Step by step:

  1. Known revenue (exclude the incomplete sale): 100 x (1 - 0) + 100 x (1 - 0.2) = 100 + 80 = 180. This is a partial total covering two of three records.
  2. Assume the missing discount is 0: 100 + 100 + 80 = 280. The assumption alone adds 100.
  3. Decision: report 180 as the known revenue with its coverage (2 of 3 sales), state that one discount is unknown, and verify the missing discount before claiming a complete total.
import pandas as pd

sales = pd.DataFrame({"gross": [100, 100, 100],
                      "discount_rate": [0.0, None, 0.2]})
sales["revenue"] = sales["gross"] * (1 - sales["discount_rate"])

known = sales["revenue"].sum()                         # NaN rows are skipped
coverage = sales["revenue"].notna().sum()
assumed = (sales["gross"] * (1 - sales["discount_rate"].fillna(0))).sum()
print(known, f"{coverage} of {len(sales)} sales", assumed, assumed - known)

Expected output: 180.0 2 of 3 sales 280.0 100.0. Notice that sum() silently skips the missing revenue, which is exactly why you must report coverage next to the total.

Habits to keep

  • Preserve the raw file and work on copies.
  • Turn every rule into explicit code and every decision into a log entry.
  • Keep unresolved records visible instead of quietly dropping them.
  • Check counts and totals after every important step.
  • Report findings with their limits, so readers know where the conclusion applies.

Keep practising with new datasets, and always be able to explain the choices you made.

Common mistakes

  • Presenting a partial total as complete. Always state how many records a total covers.
  • Treating missing as zero. A missing discount is unknown; assuming 0 can add a large, invisible amount.
  • Removing values because a rule flagged them. Statistical flags and repeated customers are prompts for checking, not proof of errors.
  • Checking only the final output. Compare before and after each important treatment, and rerun from the raw file.

Summary

  • Workflow: keep raw, explore, clean and transform with reasons, rerun and check, report findings and limits.
  • Keep, remove or fill missing values according to the question; record the effect.
  • Define uniqueness before deduplicating; correct or separate invalid values using trusted sources.
  • Outlier capping or removal needs a reason and an impact check.
  • Compare row counts, missing rates and key totals, and test whether another reasonable treatment changes your conclusion.

Flashcards · Review: The Full Workflow

8 words
WordMeaningExample
  • Full cleaning workflow

    Keep the raw copy, explore, clean and transform with reasons, rerun and check, then report findings and limits.

    Raw file to notebook to clean data, quality log and report.

  • Known total

    A total calculated only from records with complete information, reported with its coverage.

    180 from 2 of 3 sales.

  • Coverage

    The share of records that a result actually includes.

    `sales["revenue"].notna().sum()` gives 2 of 3.

  • Assumption effect

    The change in a result caused only by an assumed value, which should be stated.

    Assuming 0% discount adds 100 to the total.

  • Sensitivity check

    Testing whether a different reasonable treatment changes the conclusion.

    Compare totals with rows dropped versus filled.

  • Unresolved record

    A record kept visible in a review table or log because evidence to fix it is missing.

    A sale with an unknown discount awaiting verification.

  • Before-and-after comparison

    Comparing row counts, missing rates and key totals around each important treatment.

    Rows 108 to 100; revenue total before and after.

  • sum() skips missing

    pandas `sum()` ignores missing values by default, so a total can be silently partial.

    100 + NaN + 80 sums to 180.