Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1: Data Preparation and Model Foundations

1.7 Data Quality

Print / Save as PDF

1.7 Data Quality

After this unit you will: recognize and name the problems in raw data before training.

Key points

ProblemSign on the slide
Missing valuesBlank cells
Duplicated valuesRows that are almost identical
Wrong data formatThe value is present but written in a different style: 2025-12-10 mixed with 09/12/25, 60$ mixed with $45
OutlierA point far away from the rest and from the trend line
  • The “Variance in data” slide stresses the role of diversity: a group made up of different samples (green box) receives more attention from the model than a group of identical samples (red box).

Study notes

By the end of this lesson you can:

  • Recognise four common problems in raw data: missing values, duplicates, wrong formats and outliers.
  • Choose a basic strategy for each problem (drop, impute, deduplicate, standardise, investigate).
  • Explain what variance in a feature means and why very low or very high variance matters.
  • Use pandas to detect missing values and duplicates and to standardise inconsistent formats.

Key idea: Real-world data is messy. Before training, you must find and fix missing values, duplicate rows, inconsistent formats and suspicious outliers, because most algorithms cannot cope with them and some are badly skewed by them.

Missing values

The lecture's house table has blank cells: square_feet is missing in one row and the location in another. Gaps appear when a survey respondent skips a question, a sensor fails or data is lost during entry. Most algorithms cannot handle missing values directly, so you must decide what to do:

  • Remove rows or columns that have too many gaps, if you can afford to lose that data.
  • Impute: fill the gap with a reasonable estimate, such as the column's mean, the most common value, or a value predicted by another small model. This is usually the preferred option.

You will practise imputation with scikit-learn tools in the lab sessions.

Duplicate values

Two rows outlined in red both read 60 square feet, 4 floors, Hanoi, 3.2 billion VND. Duplicates appear when a transaction is recorded twice or when data from several sources is merged without deduplication. They bias the model: a repeated example is treated as more important or more typical than it really is. Checking for and removing duplicate rows is standard practice before training.

Wrong data format

A small table with time and cost columns mixes styles: dates such as 09/12/25 sit next to 2025-12-10, and costs appear as both $45 and 60$. A date like 09/12/25 is ambiguous: 12 September or 9 December 2025? You must pick one consistent format for each column and convert every value to it. This cleaning feels tedious, but experienced data scientists often spend more time on it than on building models.

Outliers

On a scatter plot most points lie along a diagonal trend, but one red point sits far away in the bottom right. An outlier is a value unusually different from the rest. Imagine every house is worth a few billion VND, but one row says 500 billion VND. It may be a data-entry error or a genuinely unusual property, such as a huge estate. Either way it needs attention, because algorithms such as linear and logistic regression are sensitive to outliers. You may remove it, investigate it, or use a technique that is robust to it. There is no universal rule; it depends on whether the point is valid.

Variance in data

Three people A, B and C of clearly different heights show high variance; three people of almost the same height show low variance. A feature where nearly every value is the same gives the model almost no information, because it cannot distinguish one example from another. A feature with a very wide spread can dominate the calculation of scale-sensitive algorithms. This leads directly to the next lesson on normalization and standardization.

Worked example

Detect the problems in a small table with pandas, then standardise the time and cost columns.

import numpy as np
import pandas as pd

houses = pd.DataFrame({
    "square_feet": [60, np.nan, 75, 60],
    "num_floors":  [4, 3, 5, 4],
    "location":    ["Hanoi", "Hue", None, "Hanoi"],
    "price":       [3.2, 2.1, 4.0, 3.2],
})
print(houses.isna().sum().tolist())    # missing cells per column
print(houses.duplicated().sum())       # fully repeated rows
houses = houses.drop_duplicates()

costs = pd.DataFrame({"time": ["09/12/25", "11/12/25", "2025-12-10"],
                      "cost": ["$45", "$52", "60$"]})
day_first = pd.to_datetime(costs["time"], format="%d/%m/%y", errors="coerce")
iso_style = pd.to_datetime(costs["time"], format="%Y-%m-%d", errors="coerce")
costs["time"] = day_first.fillna(iso_style)
costs["cost"] = costs["cost"].str.replace("$", "", regex=False).astype(float)
print(costs["time"].dt.strftime("%Y-%m-%d").tolist())
print(costs["cost"].tolist())

Expected output:

[1, 0, 1, 0]
1
['2025-12-09', '2025-12-11', '2025-12-10']
[45.0, 52.0, 60.0]

One cell is missing in square_feet and one in location; one row is an exact duplicate and is dropped. For the dates we chose to read dd/mm/yy (day first), which agrees with the unambiguous 2025-12-10 in the same column. After cleaning, every date has one format and every cost is a plain number.

Common mistakes

  • Deleting every row with a gap. If many rows have missing values, dropping them throws away too much data; imputation is usually better.
  • Removing every outlier automatically. Some outliers are valid, rare cases; check first, then decide.
  • Guessing the date format. Decide day-first or month-first from context and apply it consistently.
  • Forgetting near-duplicates. Exact duplicates are easy to find; also watch for rows that differ only in spacing or capitalisation.

Summary

  • Missing values: remove when affordable, otherwise impute (mean, most frequent, or a small model).
  • Duplicates bias the model and should be removed before training.
  • Inconsistent formats must be standardised to one style per column.
  • Outliers need investigation; linear and logistic regression are sensitive to them.
  • Very low variance features carry little information; very high variance features can dominate scale-sensitive algorithms.

Flashcards · 1.7 Data Quality

9 words
WordMeaningExample
  • Missing value

    An empty cell where a value should be, caused by skipped answers, sensor failures or data loss.

    A blank square_feet cell.

  • Imputation

    Filling missing values with a reasonable estimate such as the mean, the most common value, or a model's prediction.

    Fill a blank square_feet with the column mean.

  • Duplicate values

    Rows that repeat the same record; they make the model treat that example as more important.

    60, 4 floors, Hanoi, 3.2 appearing twice.

  • df.drop_duplicates()

    pandas method that removes repeated rows from a DataFrame.

    houses = houses.drop_duplicates()

  • Wrong data format

    Values written in inconsistent styles within one column, which must be standardised.

    "09/12/25" next to "2025-12-10"; "$45" next to "60$".

  • Outlier

    A data point unusually different from the rest; it may be an error or a rare valid case.

    A house listed at 500 billion VND.

  • Variance (in a feature)

    How much a feature's values differ from one another.

    Heights of very different people: high variance.

  • Low-variance feature

    A feature whose values are almost identical, so it helps little in telling examples apart.

    A column where 99% of rows say "Hanoi".

  • df.isna().sum()

    pandas expression that counts the missing cells in each column.

    houses.isna().sum()