Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Data Cleaning and Preprocessing with Python

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 2 — Clean the Data

Period 10 · Detecting Outliers with the IQR

Print / Save as PDF
Lecture video — Period 10: Detecting Outliers with the IQR

Quartiles, fences and the 1.5 IQR rule, investigating flagged values, and correcting only what the evidence supports.

Lesson overview

In this period, we will study outliers using the interquartile range, or IQR. An outlier is a value that is far from most other values in a chosen set of data. In sales data, a large quantity may be a typing mistake, but it may also be a real wholesale order. Our goal is to detect unusual values, investigate them, and compare results after a justified decision. We will use one small example throughout the lesson. Remember this point: a statistical flag does not prove that a record is wrong.

Study notes

By the end of this lesson you can:

  • Distinguish a statistical outlier from a data error.
  • Calculate Q1, Q3, the IQR and the two fences, and flag values outside them.
  • Investigate a flagged value and choose an action: retain, correct, remove or cap.
  • Apply a verified correction on a copy and compare summaries before and after.
  • Recognize when the IQR rule is unreliable (small groups, zero IQR, mixed groups).

Key idea: A statistical flag starts an investigation; it does not prove an error. Detect, investigate, decide, validate, and let evidence, not the rule, decide the final value.

Outliers versus errors

A statistical outlier is unusual relative to a reference group. A data error is incorrect according to reliable evidence or a business rule. The ideas overlap but are not the same:

  • an order of 30 items may be unusual in a retail shop and normal for a wholesale customer;
  • an order entered as 3,000 items may be a typing error, but you need the invoice to confirm it;
  • a value inside the normal range can still be wrong.

Detection supports investigation; it cannot replace it.

The IQR rule

Q1 is the 25th percentile and Q3 the 75th; IQR = Q3 - Q1 is the spread of the middle half.

  • Lower fence = Q1 - 1.5 x IQR
  • Upper fence = Q3 + 1.5 x IQR

A value is flagged if it is strictly below the lower fence or strictly above the upper fence; a value exactly on a fence is not flagged. The factor 1.5 is a common screening rule, not a business rule.

The lecture uses nine sorted quantities. With linear interpolation, Q1 = 4 and Q3 = 8, so IQR = 4, the lower fence is -2 and the upper fence is 14. Only 30 is outside. A negative lower fence does not mean negative quantities are acceptable; business validity checks stay separate, and here each row is a completed sale with a positive quantity.

Investigating and deciding

After detection, ask what the value represents: check the invoice, the units and the customer type. A large value may belong to a different but valid group; mixing retail and wholesale orders can make wholesale orders look unusual. Group-based checks help if each group has enough observations. Be careful when Q1 equals Q3: the IQR is 0, the fences collapse to one value, and many valid values may be flagged.

ActionWhen
RetainThe source confirms the value is real
CorrectReliable evidence gives the true value; keep the original too
RemoveConfirmed invalid, cannot be corrected, and should be excluded
CapA chosen analytical limit; it changes data and must be explained; it is not a correction

When evidence is incomplete, keep the value and mark it for review.

Comparing before and after

In the lecture, the invoice confirms that the true quantity is 3 and that 30 was a typing error. On a copy, replace 30 with 3 at index 8 (the ninth record). The correction is justified by the invoice, not by the IQR rule; the rule never tells you what the right number is. In real data, select the record by its stable business key and log the old value, new value, evidence and reason.

StatisticBeforeAfter
count99
sum7447
mean8.225.22
median65
max309

Total quantity falls by 27 units. Smaller numbers do not prove the data are better; the evidence does. For a consistent comparison, keep the original fences, or clearly document new thresholds if you recalculate them.

Worked example

The nine quantities below are one set that reproduces all the lecture's numbers.

import pandas as pd

qty = pd.Series([2, 3, 4, 5, 6, 7, 8, 9, 30], name="quantity")
q1, q3 = qty.quantile([0.25, 0.75], interpolation="linear")
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
flag = (qty < lower) | (qty > upper)
print(q1, q3, iqr, lower, upper)          # 4.0 8.0 4.0 -2.0 14.0
print(qty[flag].tolist())                 # [30]

corrected = qty.copy()
corrected.loc[8] = 3                      # invoice confirms 3; 30 was a typing error
stats = ["count", "sum", "mean", "median", "max"]
compare = pd.DataFrame({"before": qty.agg(stats), "after": corrected.agg(stats)})
assert len(corrected) == len(qty) and (corrected > 0).all()
print(compare.round(2))

Expected output: fences -2.0 and 14.0, flagged value 30, then a table matching the one above (sum 74 to 47, mean 8.22 to 5.22, median 6 to 5). If the invoice had confirmed 30, the right action would be to retain it and document why it was flagged.

In the course notebook the same idea runs on quantity_num of the accepted rows, and a practice task applies the rule to unit_price_num within each product_category, because prices differ a lot between categories.

Common mistakes

  • Deleting every flagged value. A flag is a question; a real wholesale order must stay.
  • Letting the rule pick the correction. Only reliable evidence such as an invoice can supply the true value.
  • Reading a negative lower fence as permission. Business rules still require positive quantities.
  • Applying the rule to tiny or mixed groups. A zero IQR or a mix of retail and wholesale orders produces misleading flags.

Summary

  • Outliers are unusual; errors are wrong by evidence. They overlap but are different.
  • Fences: Q1 - 1.5 x IQR and Q3 + 1.5 x IQR; values strictly outside are flagged.
  • Investigate invoices, units and customer type before acting.
  • Retain, correct, remove or cap only with a stated reason; keep the original value.
  • Compare count, sum, mean, median and max before and after, and log every change.

Flashcards · Detecting Outliers with the IQR

9 words
WordMeaningExample
  • Statistical outlier

    A value far from most other values in a chosen reference group.

    30 items when most orders are 2-9 items.

  • Data error

    A value shown to be incorrect by reliable evidence or a business rule.

    Invoice shows 3 items, data say 30.

  • IQR fences

    Limits at Q1 - 1.5 x IQR and Q3 + 1.5 x IQR; values strictly outside are flagged.

    Q1 4, Q3 8 gives fences -2 and 14.

  • Retain

    Keeping a flagged value because the source confirms it is real, and documenting why it was flagged.

    A confirmed wholesale order of 30 items.

  • Correct

    Replacing a value only when reliable evidence gives the true value, while preserving the original.

    `corrected.loc[8] = 3` with the invoice logged.

  • Capping

    Limiting values to a chosen bound for analysis; it changes the data and is not a correction.

    Set quantities above 14 to 14, and explain why.

  • Zero IQR

    When Q1 equals Q3 the fences collapse, so many valid values may be flagged.

    A small group where most quantities are 1.

  • Before-and-after summary

    A table comparing key statistics before and after a treatment to show its effect.

    Sum 74 to 47, mean 8.22 to 5.22.

  • Series.quantile()

    Returns values at given percentiles, using linear interpolation by default.

    `qty.quantile([0.25, 0.75])`

Practice for Session 4

Open notebooks/Session4_Outliers_Transform_Report.ipynb from the practice pack (dc_practice_pack.zip, attached to the Course Overview lesson) together with sales_transactions_raw.csv. Run it in Google Colab or Jupyter from top to bottom and complete the Your turn exercises.