You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
In this period, we will study outliers using the interquartile range, or IQR. An outlier is a value that is far from most other values in a chosen set of data. In sales data, a large quantity may be a typing mistake, but it may also be a real wholesale order. Our goal is to detect unusual values, investigate them, and compare results after a justified decision. We will use one small example throughout the lesson. Remember this point: a statistical flag does not prove that a record is wrong.
By the end of this lesson you can:
Key idea: A statistical flag starts an investigation; it does not prove an error. Detect, investigate, decide, validate, and let evidence, not the rule, decide the final value.
A statistical outlier is unusual relative to a reference group. A data error is incorrect according to reliable evidence or a business rule. The ideas overlap but are not the same:
Detection supports investigation; it cannot replace it.
Q1 is the 25th percentile and Q3 the 75th; IQR = Q3 - Q1 is the spread of the middle half.
A value is flagged if it is strictly below the lower fence or strictly above the upper fence; a value exactly on a fence is not flagged. The factor 1.5 is a common screening rule, not a business rule.
The lecture uses nine sorted quantities. With linear interpolation, Q1 = 4 and Q3 = 8, so IQR = 4, the lower fence is -2 and the upper fence is 14. Only 30 is outside. A negative lower fence does not mean negative quantities are acceptable; business validity checks stay separate, and here each row is a completed sale with a positive quantity.
After detection, ask what the value represents: check the invoice, the units and the customer type. A large value may belong to a different but valid group; mixing retail and wholesale orders can make wholesale orders look unusual. Group-based checks help if each group has enough observations. Be careful when Q1 equals Q3: the IQR is 0, the fences collapse to one value, and many valid values may be flagged.
| Action | When |
|---|---|
| Retain | The source confirms the value is real |
| Correct | Reliable evidence gives the true value; keep the original too |
| Remove | Confirmed invalid, cannot be corrected, and should be excluded |
| Cap | A chosen analytical limit; it changes data and must be explained; it is not a correction |
When evidence is incomplete, keep the value and mark it for review.
In the lecture, the invoice confirms that the true quantity is 3 and that 30 was a typing error. On a copy, replace 30 with 3 at index 8 (the ninth record). The correction is justified by the invoice, not by the IQR rule; the rule never tells you what the right number is. In real data, select the record by its stable business key and log the old value, new value, evidence and reason.
| Statistic | Before | After |
|---|---|---|
| count | 9 | 9 |
| sum | 74 | 47 |
| mean | 8.22 | 5.22 |
| median | 6 | 5 |
| max | 30 | 9 |
Total quantity falls by 27 units. Smaller numbers do not prove the data are better; the evidence does. For a consistent comparison, keep the original fences, or clearly document new thresholds if you recalculate them.
The nine quantities below are one set that reproduces all the lecture's numbers.
import pandas as pd
qty = pd.Series([2, 3, 4, 5, 6, 7, 8, 9, 30], name="quantity")
q1, q3 = qty.quantile([0.25, 0.75], interpolation="linear")
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
flag = (qty < lower) | (qty > upper)
print(q1, q3, iqr, lower, upper) # 4.0 8.0 4.0 -2.0 14.0
print(qty[flag].tolist()) # [30]
corrected = qty.copy()
corrected.loc[8] = 3 # invoice confirms 3; 30 was a typing error
stats = ["count", "sum", "mean", "median", "max"]
compare = pd.DataFrame({"before": qty.agg(stats), "after": corrected.agg(stats)})
assert len(corrected) == len(qty) and (corrected > 0).all()
print(compare.round(2))
Expected output: fences -2.0 and 14.0, flagged value 30, then a table matching the one above (sum 74 to 47, mean 8.22 to 5.22, median 6 to 5). If the invoice had confirmed 30, the right action would be to retain it and document why it was flagged.
In the course notebook the same idea runs on quantity_num of the accepted rows, and a practice task applies the rule to unit_price_num within each product_category, because prices differ a lot between categories.
Flashcards · Detecting Outliers with the IQR
9 wordsA value far from most other values in a chosen reference group.
A value shown to be incorrect by reliable evidence or a business rule.
Limits at Q1 - 1.5 x IQR and Q3 + 1.5 x IQR; values strictly outside are flagged.
Keeping a flagged value because the source confirms it is real, and documenting why it was flagged.
Replacing a value only when reliable evidence gives the true value, while preserving the original.
Limiting values to a chosen bound for analysis; it changes the data and is not a correction.
When Q1 equals Q3 the fences collapse, so many valid values may be flagged.
A table comparing key statistics before and after a treatment to show its effect.
Returns values at given percentiles, using linear interpolation by default.
Open notebooks/Session4_Outliers_Transform_Report.ipynb from the practice pack (dc_practice_pack.zip, attached to the Course Overview lesson) together with sales_transactions_raw.csv. Run it in Google Colab or Jupyter from top to bottom and complete the Your turn exercises.