Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Data Cleaning and Preprocessing with Python

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1 — Explore and Understand the Data

Period 1 · Data Quality and the Cleaning Workflow

Print / Save as PDF
Lecture video — Period 1: Data Quality and the Cleaning Workflow

How simple errors in a small sales table distort a report, and the main steps from raw data to a reliable result.

Lesson overview

In the first period, we will look at data quality before we write any Python code. We will use a small sales table to see how simple errors can affect a report. By the end, you should be able to spot common problems, describe the main steps from raw data to a report, and explain why we must keep the original file.

Study notes

By the end of this lesson you can:

  • Define data quality as "fit for the task" and name the checks it involves (accuracy, completeness, consistency, validity, uniqueness).
  • Explain why a valid value is not always an accurate value.
  • Describe the cleaning workflow from the original file to a checked report.
  • Spot at least three problems in a small sales table and say what evidence you need before changing them.
  • Explain why the raw file must stay unchanged.

Key idea: Check the data before you trust the report. A calculation can be perfectly correct for the rows it receives and still give a wrong business result if the rows themselves are wrong.

What data quality means

Data quality means that the data are suitable for the task. For a sales report, we ask several separate questions:

QuestionQuality dimensionExample problem
Do the values match real sales?AccuracyA price typed as 50,000 when the receipt says 55,000
Are the important fields filled in?CompletenessA row with no quantity
Do labels follow one format?ConsistencyNorth and north in the same column
Do values follow business rules?ValidityA negative quantity in a table of completed sales
Is each record recorded once?UniquenessThe same sale entered twice

Validity and accuracy are different. A price of 50,000 dong may follow the required format (a number, not negative) and still be wrong. Only the source record, such as a receipt or invoice, can confirm accuracy. So a valid value is not always an accurate value.

Why errors matter before any chart

Consider a customer who buys two items at 50,000 dong each. The correct sales amount is 100,000 dong. If the same sale is recorded twice, the report adds both rows and shows 200,000 dong. Python did nothing wrong: the sum is correct for the rows it was given. The business result is still wrong. This is why we check data before making charts and tables.

Be careful with the opposite mistake too. A repeated order number alone does not prove a duplicate, because one order may contain several products. Before you call two rows duplicates, you must understand what one row represents. In the lecture's small table, one row is one order line, so the pair (order number, line number) should be unique. In the course dataset sales_transactions_raw.csv, one row is one transaction for one product category, and the key is transaction_id.

The cleaning workflow

The lecture describes a workflow that starts from the original file:

  1. Keep the original file unchanged.
  2. Inspect the data and understand what each row means.
  3. Clean confirmed problems using clear rules.
  4. Prepare useful fields (for example a sales amount) and build summary tables.
  5. Check the results before reporting, for example by comparing row counts and total sales before and after processing.
  6. Report the findings and any unresolved issues.

The workflow is a cycle, not a straight line. If a check shows an unexpected change, go back to an earlier step.

Keeping the raw file and recording decisions

We keep the original file so that we can trace our work and start again if necessary. A good project uses separate locations for raw data, the working notebook and the outputs. The notebook records each check and the reason for each change, and we keep a list of unresolved issues.

Saving a copy is only the first step. A useful workflow can run again from the raw file and produce the same result. Never overwrite the original file with the cleaned version. For every action, record three things: the rule, the change, and its effect on the data (rows and totals).

Worked example

The lecture shows a small order-line table. Quantities must be positive (returns are stored in a separate table) and a dash means a missing value. Suppose the table looks like this:

RowOrderLineRegionQuantityUnit price
1O11North250,000
2O21South180,000
3O31North-60,000
4O41north-140,000
5O11North250,000

Step by step:

  1. Labels. North and north probably describe the same region. Confirm this with the data dictionary or data owner, then use one label.
  2. Missing value. Row 3 has no quantity, so we cannot calculate its sales amount yet. Replacing it with zero would be an unsupported assumption: we would be claiming that nothing was sold.
  3. Business rule. Row 4 has a negative quantity, which breaks the positive-quantity rule. It might be a return stored in the wrong table, or a typing error. Check before changing it.
  4. Repeated key. Rows 1 and 5 are identical and repeat the same (order, line) key. Verify that this is a recording error before removing the copy.

For each finding, write the rule, the planned change and its effect, for example: "Rows 1 and 5 duplicate key O1-1; if confirmed, remove row 5; row count 5 to 4, total sales falls by 100,000 dong."

Common mistakes

  • Treating a valid value as correct. A number in the right format can still be wrong. Accuracy needs the source record.
  • Filling a missing quantity with zero. Zero means "no items sold", which is a claim. Use evidence or an agreed treatment instead.
  • Deleting rows because an order number repeats. One order can contain several products. Define the key first, then check.
  • Overwriting the raw file. If the cleaned file replaces the original, you cannot trace or redo your work.

Summary

  • Data quality means data are fit for the task: accurate, complete, consistent, valid and not repeated.
  • A valid value is not always accurate; you need the source record to confirm accuracy.
  • Correct code on wrong rows gives a wrong business result, so check the data before charts.
  • Know what one row represents before you decide what a duplicate is.
  • Follow the workflow: keep raw, inspect, clean with rules, prepare, check, report; loop back when needed.
  • Record the rule, the change and its effect for every decision.

Flashcards · Data Quality and the Cleaning Workflow

9 words
WordMeaningExample
  • Data quality

    The degree to which data are suitable for a specific task, such as producing a correct sales report.

    Sales data are high quality for a report if values are accurate, complete and consistent.

  • Accuracy

    Whether a recorded value matches the real-world fact, which can only be confirmed with the source record.

    Compare a recorded price of 50,000 dong with the sales receipt.

  • Validity

    Whether a value follows the required format and business rules, regardless of whether it is true.

    Quantity must be a positive whole number.

  • Consistency

    Whether the same thing is always written the same way across records.

    Using only `North`, never `north` or `N.`.

  • Business key

    The column or combination of columns that should identify one record uniquely, based on what one row represents.

    `transaction_id` in the course dataset; (order, line) for order lines.

  • Duplicate record

    A row that records the same business event as another row, confirmed by checking the key and values.

    Rows 1 and 5 both record order O1, line 1, quantity 2.

  • Raw file

    The original data file, kept unchanged so that every step can be traced and repeated.

    Keep `sales_transactions_raw.csv` read-only; save cleaned data elsewhere.

  • Cleaning workflow

    The cycle of keeping raw data, inspecting, cleaning with rules, preparing fields, checking and reporting.

    Compare row counts and total sales before and after cleaning.

  • Decision log

    A record of each cleaning rule, the change made and its effect on rows and totals.

    Removed 1 duplicate row; rows 5 to 4; sales down 100,000 dong.

Practice for Session 1

Open notebooks/Session1_Examine_and_Profile.ipynb from the practice pack (dc_practice_pack.zip, attached to the Course Overview lesson) together with sales_transactions_raw.csv. Run it in Google Colab or Jupyter from top to bottom and complete the Your turn exercises.