You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
In the first period, we will look at data quality before we write any Python code. We will use a small sales table to see how simple errors can affect a report. By the end, you should be able to spot common problems, describe the main steps from raw data to a report, and explain why we must keep the original file.
By the end of this lesson you can:
Key idea: Check the data before you trust the report. A calculation can be perfectly correct for the rows it receives and still give a wrong business result if the rows themselves are wrong.
Data quality means that the data are suitable for the task. For a sales report, we ask several separate questions:
| Question | Quality dimension | Example problem |
|---|---|---|
| Do the values match real sales? | Accuracy | A price typed as 50,000 when the receipt says 55,000 |
| Are the important fields filled in? | Completeness | A row with no quantity |
| Do labels follow one format? | Consistency | North and north in the same column |
| Do values follow business rules? | Validity | A negative quantity in a table of completed sales |
| Is each record recorded once? | Uniqueness | The same sale entered twice |
Validity and accuracy are different. A price of 50,000 dong may follow the required format (a number, not negative) and still be wrong. Only the source record, such as a receipt or invoice, can confirm accuracy. So a valid value is not always an accurate value.
Consider a customer who buys two items at 50,000 dong each. The correct sales amount is 100,000 dong. If the same sale is recorded twice, the report adds both rows and shows 200,000 dong. Python did nothing wrong: the sum is correct for the rows it was given. The business result is still wrong. This is why we check data before making charts and tables.
Be careful with the opposite mistake too. A repeated order number alone does not prove a duplicate, because one order may contain several products. Before you call two rows duplicates, you must understand what one row represents. In the lecture's small table, one row is one order line, so the pair (order number, line number) should be unique. In the course dataset sales_transactions_raw.csv, one row is one transaction for one product category, and the key is transaction_id.
The lecture describes a workflow that starts from the original file:
The workflow is a cycle, not a straight line. If a check shows an unexpected change, go back to an earlier step.
We keep the original file so that we can trace our work and start again if necessary. A good project uses separate locations for raw data, the working notebook and the outputs. The notebook records each check and the reason for each change, and we keep a list of unresolved issues.
Saving a copy is only the first step. A useful workflow can run again from the raw file and produce the same result. Never overwrite the original file with the cleaned version. For every action, record three things: the rule, the change, and its effect on the data (rows and totals).
The lecture shows a small order-line table. Quantities must be positive (returns are stored in a separate table) and a dash means a missing value. Suppose the table looks like this:
| Row | Order | Line | Region | Quantity | Unit price |
|---|---|---|---|---|---|
| 1 | O1 | 1 | North | 2 | 50,000 |
| 2 | O2 | 1 | South | 1 | 80,000 |
| 3 | O3 | 1 | North | - | 60,000 |
| 4 | O4 | 1 | north | -1 | 40,000 |
| 5 | O1 | 1 | North | 2 | 50,000 |
Step by step:
North and north probably describe the same region. Confirm this with the data dictionary or data owner, then use one label.For each finding, write the rule, the planned change and its effect, for example: "Rows 1 and 5 duplicate key O1-1; if confirmed, remove row 5; row count 5 to 4, total sales falls by 100,000 dong."
Flashcards · Data Quality and the Cleaning Workflow
9 wordsThe degree to which data are suitable for a specific task, such as producing a correct sales report.
Whether a recorded value matches the real-world fact, which can only be confirmed with the source record.
Whether a value follows the required format and business rules, regardless of whether it is true.
Whether the same thing is always written the same way across records.
The column or combination of columns that should identify one record uniquely, based on what one row represents.
A row that records the same business event as another row, confirmed by checking the key and values.
The original data file, kept unchanged so that every step can be traced and repeated.
The cycle of keeping raw data, inspecting, cleaning with rules, preparing fields, checking and reporting.
A record of each cleaning rule, the change made and its effect on rows and totals.
Open notebooks/Session1_Examine_and_Profile.ipynb from the practice pack (dc_practice_pack.zip, attached to the Course Overview lesson) together with sales_transactions_raw.csv. Run it in Google Colab or Jupyter from top to bottom and complete the Your turn exercises.