Period 1 · Data Quality and the Cleaning Workflow
[video] Lecture video — Period 1: Data Quality and the Cleaning Workflow — /api/lesson-media/videos/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794846239409-479d5310f44434e0.mp4
[markdown] content available only on the platform
[markdown] content available only on the platform
Flashcards · Data Quality and the Cleaning Workflow
Data quality — The degree to which data are suitable for a specific task, such as producing a correct sales report.
Sales data are high quality for a report if values are accurate, complete and consistent.
Accuracy — Whether a recorded value matches the real-world fact, which can only be confirmed with the source record.
Compare a recorded price of 50,000 dong with the sales receipt.
Validity — Whether a value follows the required format and business rules, regardless of whether it is true.
Quantity must be a positive whole number.
Consistency — Whether the same thing is always written the same way across records.
Using only `North`, never `north` or `N.`.
Business key — The column or combination of columns that should identify one record uniquely, based on what one row represents.
`transaction_id` in the course dataset; (order, line) for order lines.
Duplicate record — A row that records the same business event as another row, confirmed by checking the key and values.
Rows 1 and 5 both record order O1, line 1, quantity 2.
Raw file — The original data file, kept unchanged so that every step can be traced and repeated.
Keep `sales_transactions_raw.csv` read-only; save cleaned data elsewhere.
Cleaning workflow — The cycle of keeping raw data, inspecting, cleaning with rules, preparing fields, checking and reporting.
Compare row counts and total sales before and after cleaning.
Decision log — A record of each cleaning rule, the change made and its effect on rows and totals.
Removed 1 duplicate row; rows 5 to 4; sales down 100,000 dong.
[pdf] Lecture slides — Period 1 (PDF) — /api/lesson-media/pdfs/4d3653ff-a3ca-4f63-bf8e-bb93c553e599-1794873691236-ee9e30edde19f791.pdf
[markdown] content available only on the platform
