[video] /api/lesson-media/videos/9b417aa2-5e9f-4ae3-a9db-2a3b7f03f372-1794891206641-7af8389cdac9a5d2.mp4
[markdown] content available only on the platform
[markdown] content available only on the platform
Flashcards · 1.7 Data Quality
Missing value — An empty cell where a value should be, caused by skipped answers, sensor failures or data loss.
A blank square_feet cell.
Imputation — Filling missing values with a reasonable estimate such as the mean, the most common value, or a model's prediction.
Fill a blank square_feet with the column mean.
Duplicate values — Rows that repeat the same record; they make the model treat that example as more important.
60, 4 floors, Hanoi, 3.2 appearing twice.
df.drop_duplicates() — pandas method that removes repeated rows from a DataFrame.
houses = houses.drop_duplicates()
Wrong data format — Values written in inconsistent styles within one column, which must be standardised.
"09/12/25" next to "2025-12-10"; "$45" next to "60$".
Outlier — A data point unusually different from the rest; it may be an error or a rare valid case.
A house listed at 500 billion VND.
Variance (in a feature) — How much a feature's values differ from one another.
Heights of very different people: high variance.
Low-variance feature — A feature whose values are almost identical, so it helps little in telling examples apart.
A column where 99% of rows say "Hanoi".
df.isna().sum() — pandas expression that counts the missing cells in each column.
houses.isna().sum()