You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
After this unit you will: recognize and name the problems in raw data before training.
| Problem | Sign on the slide |
|---|---|
| Missing values | Blank cells |
| Duplicated values | Rows that are almost identical |
| Wrong data format | The value is present but written in a different style: 2025-12-10 mixed with 09/12/25, 60$ mixed with $45 |
| Outlier | A point far away from the rest and from the trend line |
By the end of this lesson you can:
Key idea: Real-world data is messy. Before training, you must find and fix missing values, duplicate rows, inconsistent formats and suspicious outliers, because most algorithms cannot cope with them and some are badly skewed by them.
The lecture's house table has blank cells: square_feet is missing in one row and the location in another. Gaps appear when a survey respondent skips a question, a sensor fails or data is lost during entry. Most algorithms cannot handle missing values directly, so you must decide what to do:
You will practise imputation with scikit-learn tools in the lab sessions.
Two rows outlined in red both read 60 square feet, 4 floors, Hanoi, 3.2 billion VND. Duplicates appear when a transaction is recorded twice or when data from several sources is merged without deduplication. They bias the model: a repeated example is treated as more important or more typical than it really is. Checking for and removing duplicate rows is standard practice before training.
A small table with time and cost columns mixes styles: dates such as 09/12/25 sit next to 2025-12-10, and costs appear as both $45 and 60$. A date like 09/12/25 is ambiguous: 12 September or 9 December 2025? You must pick one consistent format for each column and convert every value to it. This cleaning feels tedious, but experienced data scientists often spend more time on it than on building models.
On a scatter plot most points lie along a diagonal trend, but one red point sits far away in the bottom right. An outlier is a value unusually different from the rest. Imagine every house is worth a few billion VND, but one row says 500 billion VND. It may be a data-entry error or a genuinely unusual property, such as a huge estate. Either way it needs attention, because algorithms such as linear and logistic regression are sensitive to outliers. You may remove it, investigate it, or use a technique that is robust to it. There is no universal rule; it depends on whether the point is valid.
Three people A, B and C of clearly different heights show high variance; three people of almost the same height show low variance. A feature where nearly every value is the same gives the model almost no information, because it cannot distinguish one example from another. A feature with a very wide spread can dominate the calculation of scale-sensitive algorithms. This leads directly to the next lesson on normalization and standardization.
Detect the problems in a small table with pandas, then standardise the time and cost columns.
import numpy as np
import pandas as pd
houses = pd.DataFrame({
"square_feet": [60, np.nan, 75, 60],
"num_floors": [4, 3, 5, 4],
"location": ["Hanoi", "Hue", None, "Hanoi"],
"price": [3.2, 2.1, 4.0, 3.2],
})
print(houses.isna().sum().tolist()) # missing cells per column
print(houses.duplicated().sum()) # fully repeated rows
houses = houses.drop_duplicates()
costs = pd.DataFrame({"time": ["09/12/25", "11/12/25", "2025-12-10"],
"cost": ["$45", "$52", "60$"]})
day_first = pd.to_datetime(costs["time"], format="%d/%m/%y", errors="coerce")
iso_style = pd.to_datetime(costs["time"], format="%Y-%m-%d", errors="coerce")
costs["time"] = day_first.fillna(iso_style)
costs["cost"] = costs["cost"].str.replace("$", "", regex=False).astype(float)
print(costs["time"].dt.strftime("%Y-%m-%d").tolist())
print(costs["cost"].tolist())
Expected output:
[1, 0, 1, 0]
1
['2025-12-09', '2025-12-11', '2025-12-10']
[45.0, 52.0, 60.0]
One cell is missing in square_feet and one in location; one row is an exact duplicate and is dropped. For the dates we chose to read dd/mm/yy (day first), which agrees with the unambiguous 2025-12-10 in the same column. After cleaning, every date has one format and every cost is a plain number.
Flashcards · 1.7 Data Quality
9 wordsAn empty cell where a value should be, caused by skipped answers, sensor failures or data loss.
Filling missing values with a reasonable estimate such as the mean, the most common value, or a model's prediction.
Rows that repeat the same record; they make the model treat that example as more important.
pandas method that removes repeated rows from a DataFrame.
Values written in inconsistent styles within one column, which must be standardised.
A data point unusually different from the rest; it may be an error or a rare valid case.
How much a feature's values differ from one another.
A feature whose values are almost identical, so it helps little in telling examples apart.
pandas expression that counts the missing cells in each column.