You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
In Period 3, we checked data quality. We now use descriptive statistics to understand numeric columns. We will answer two questions: what is a typical value, and how much do values vary? Our examples use quantity and unit price in a small sales table. By the end, you should be able to calculate the main measures in Python and explain what they mean. A useful report needs both a number and a clear interpretation. We will combine the theory with a short calculation exercise.
By the end of this lesson you can:
ddof convention you used.agg() and quantile().Key idea: A useful description of a numeric column needs both a typical value and a measure of spread, with units and the number of observations, plus a clear interpretation.
The lecture uses five illustrative transactions. quantity is in items and unit_price is in thousand Vietnamese dong per item. Both columns are numeric and complete, and each transaction has equal weight. So the mean unit price is an average across transactions, not a price weighted by the number of items sold.
The quantities are 2, 3, 3, 4 and 20.
The quantity of 20 is much larger than the others, but that alone does not make it an error. Compare with the set 2, 3, 3, 4, 5: mean 3.4, median 3. Changing only the last value to 20 raises the mean to 6.4 while the median stays at 3. The median resists very large values, but the large order still counts in total sales volume, so the mean is informative too. Report both when they tell different parts of the story, and do not treat this as a reason to remove the order.
Center does not show how widely values spread. For variance, subtract the mean from each value, square the differences and add them. For quantity this total is 233.2.
| Convention | Denominator | Variance | Standard deviation |
|---|---|---|---|
Sample, ddof=1 (pandas default) | n - 1 = 4 | 58.3 | about 7.64 items |
Population, ddof=0 | n = 5 | 46.64 | about 6.83 items |
Variance has squared units (items squared); standard deviation is in the original unit, which makes it easier to interpret. Because differences are squared, a value far from the mean has a strong effect on both. Both conventions can be correct; state your choice and use it consistently.
Quantiles describe positions in sorted data. Q1 is the 25th percentile, Q2 is the median, and Q3 is the 75th percentile. For the five quantities, Q1 = 3 and Q3 = 4, so the interquartile range IQR = Q3 - Q1 = 1 item. It describes the width of the central part of the distribution. With small samples and repeated values, do not expect exactly half of the rows to lie between the quartiles. pandas uses linear interpolation by default; state this so others can reproduce your numbers.
Quantity has mean 6.4 and median 3: the large order pulls the mean up, while the IQR of 1 item shows that the central quantities are close together. For unit price, the lecture reports mean 12.8 and median 12 thousand dong per item, with an IQR of 2.
Do not conclude that quantity "varies more" than price by comparing their standard deviations: the units differ (items versus thousand dong). Different product categories may also explain price differences. In real data, statistics skip missing values, so check count for each column; a missing value should never silently become zero.
import pandas as pd
sales = pd.DataFrame({"quantity": [2, 3, 3, 4, 20]})
summary = sales[["quantity"]].agg(["count", "mean", "median", "var", "std"]).T
q = sales[["quantity"]].quantile([0.25, 0.75]).T # linear interpolation (default)
summary["Q1"], summary["Q3"] = q[0.25], q[0.75]
summary["IQR"] = summary["Q3"] - summary["Q1"]
print(summary.round(2))
print(round(sales["quantity"].std(ddof=0), 2)) # population convention
Expected output: count 5, mean 6.4, median 3.0, var 58.3, std 7.64, Q1 3.0, Q3 4.0, IQR 1.0; the population standard deviation is 6.83. round() only changes the display, not the data.
Quick check from the lecture: if the 20 were 5, the mean would fall to 3.4, the median would stay at 3, the sample standard deviation would fall to about 1.14 and the IQR would stay at 1. This is a comparison scenario, not an instruction to change the original value.
ddof. pandas var() and std() use ddof=1; a calculator or another library may use ddof=0. Say which you used.ddof=1 by default; ddof=0 describes a complete population.Flashcards · Describing Numeric Columns
9 wordsThe sum of the values divided by their number; sensitive to very large or small values.
The middle value of the sorted data, or the average of the two middle values when n is even.
The average squared distance from the mean, measured in squared units.
The square root of the variance, expressed in the original unit of the data.
Delta degrees of freedom: the denominator is n minus ddof; pandas uses 1 by default.
The values at the 25th and 75th percentiles of the sorted data.
Interquartile range, Q3 minus Q1: the width of the middle half of the values.
Applies several summary functions at once and returns them in one table.
The default pandas method for quantiles that fall between two observed values.
Open notebooks/Session2_Explore_EDA.ipynb from the practice pack (dc_practice_pack.zip, attached to the Course Overview lesson) together with sales_transactions_raw.csv. Run it in Google Colab or Jupyter from top to bottom and complete the Your turn exercises.