Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Data Cleaning and Preprocessing with Python

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1 — Explore and Understand the Data

Period 4 · Describing Numeric Columns

Print / Save as PDF
Lecture video — Period 4: Describing Numeric Columns

Mean and median, variance and standard deviation, quartiles and the IQR, and reporting statistics with their units and counts.

Lesson overview

In Period 3, we checked data quality. We now use descriptive statistics to understand numeric columns. We will answer two questions: what is a typical value, and how much do values vary? Our examples use quantity and unit price in a small sales table. By the end, you should be able to calculate the main measures in Python and explain what they mean. A useful report needs both a number and a clear interpretation. We will combine the theory with a short calculation exercise.

Study notes

By the end of this lesson you can:

  • Calculate and interpret the mean and the median, and explain why they can differ.
  • Calculate variance and standard deviation and state which ddof convention you used.
  • Find Q1, Q3 and the IQR with pandas and explain what they describe.
  • Produce one summary table with agg() and quantile().
  • Report statistics with center, spread, units and observation count.

Key idea: A useful description of a numeric column needs both a typical value and a measure of spread, with units and the number of observations, plus a clear interpretation.

The example and measures of center

The lecture uses five illustrative transactions. quantity is in items and unit_price is in thousand Vietnamese dong per item. Both columns are numeric and complete, and each transaction has equal weight. So the mean unit price is an average across transactions, not a price weighted by the number of items sold.

The quantities are 2, 3, 3, 4 and 20.

  • Mean: add all values and divide by the count. 32 / 5 = 6.4 items per transaction. A mean can be a decimal even though each quantity is a whole number.
  • Median: sort the values and take the middle one. The third of five values is 3. With six values, average the third and fourth.

The quantity of 20 is much larger than the others, but that alone does not make it an error. Compare with the set 2, 3, 3, 4, 5: mean 3.4, median 3. Changing only the last value to 20 raises the mean to 6.4 while the median stays at 3. The median resists very large values, but the large order still counts in total sales volume, so the mean is informative too. Report both when they tell different parts of the story, and do not treat this as a reason to remove the order.

Variance and standard deviation

Center does not show how widely values spread. For variance, subtract the mean from each value, square the differences and add them. For quantity this total is 233.2.

ConventionDenominatorVarianceStandard deviation
Sample, ddof=1 (pandas default)n - 1 = 458.3about 7.64 items
Population, ddof=0n = 546.64about 6.83 items

Variance has squared units (items squared); standard deviation is in the original unit, which makes it easier to interpret. Because differences are squared, a value far from the mean has a strong effect on both. Both conventions can be correct; state your choice and use it consistently.

Quartiles and the IQR

Quantiles describe positions in sorted data. Q1 is the 25th percentile, Q2 is the median, and Q3 is the 75th percentile. For the five quantities, Q1 = 3 and Q3 = 4, so the interquartile range IQR = Q3 - Q1 = 1 item. It describes the width of the central part of the distribution. With small samples and repeated values, do not expect exactly half of the rows to lie between the quartiles. pandas uses linear interpolation by default; state this so others can reproduce your numbers.

Interpreting the summary

Quantity has mean 6.4 and median 3: the large order pulls the mean up, while the IQR of 1 item shows that the central quantities are close together. For unit price, the lecture reports mean 12.8 and median 12 thousand dong per item, with an IQR of 2.

Do not conclude that quantity "varies more" than price by comparing their standard deviations: the units differ (items versus thousand dong). Different product categories may also explain price differences. In real data, statistics skip missing values, so check count for each column; a missing value should never silently become zero.

Worked example

import pandas as pd

sales = pd.DataFrame({"quantity": [2, 3, 3, 4, 20]})

summary = sales[["quantity"]].agg(["count", "mean", "median", "var", "std"]).T
q = sales[["quantity"]].quantile([0.25, 0.75]).T     # linear interpolation (default)
summary["Q1"], summary["Q3"] = q[0.25], q[0.75]
summary["IQR"] = summary["Q3"] - summary["Q1"]
print(summary.round(2))

print(round(sales["quantity"].std(ddof=0), 2))       # population convention

Expected output: count 5, mean 6.4, median 3.0, var 58.3, std 7.64, Q1 3.0, Q3 4.0, IQR 1.0; the population standard deviation is 6.83. round() only changes the display, not the data.

Quick check from the lecture: if the 20 were 5, the mean would fall to 3.4, the median would stay at 3, the sample standard deviation would fall to about 1.14 and the IQR would stay at 1. This is a comparison scenario, not an instruction to change the original value.

Common mistakes

  • Reporting only the mean. One large value can pull it far from the typical transaction; show the median and spread as well.
  • Ignoring ddof. pandas var() and std() use ddof=1; a calculator or another library may use ddof=0. Say which you used.
  • Comparing spreads across different units. A larger standard deviation in items does not mean "more variable" than one in thousand dong.
  • Removing a large value because it changes the mean. A statistic is not evidence of an error; check the source record.

Summary

  • Mean = sum / count; median = middle of the sorted values; they react differently to large values.
  • Variance averages squared deviations; standard deviation is its square root, in the original unit.
  • pandas uses ddof=1 by default; ddof=0 describes a complete population.
  • IQR = Q3 - Q1 describes the middle half; pandas quantiles use linear interpolation.
  • Always report center, spread, units and the observation count, with an interpretation.

Flashcards · Describing Numeric Columns

9 words
WordMeaningExample
  • Mean

    The sum of the values divided by their number; sensitive to very large or small values.

    (2 + 3 + 3 + 4 + 20) / 5 = 6.4 items

  • Median

    The middle value of the sorted data, or the average of the two middle values when n is even.

    2, 3, 3, 4, 20 gives median 3.

  • Variance

    The average squared distance from the mean, measured in squared units.

    `df["quantity"].var()` gives 58.3 for the lecture data.

  • Standard deviation

    The square root of the variance, expressed in the original unit of the data.

    `df["quantity"].std()` gives about 7.64 items.

  • ddof

    Delta degrees of freedom: the denominator is n minus ddof; pandas uses 1 by default.

    `std(ddof=0)` for a complete population.

  • Quartiles (Q1, Q3)

    The values at the 25th and 75th percentiles of the sorted data.

    `df["quantity"].quantile([0.25, 0.75])` gives 3 and 4.

  • IQR

    Interquartile range, Q3 minus Q1: the width of the middle half of the values.

    4 - 3 = 1 item

  • df.agg()

    Applies several summary functions at once and returns them in one table.

    `df[cols].agg(["count", "mean", "median", "var", "std"])`

  • Linear interpolation

    The default pandas method for quantiles that fall between two observed values.

    `quantile(0.25, interpolation="linear")`

Practice for Session 2

Open notebooks/Session2_Explore_EDA.ipynb from the practice pack (dc_practice_pack.zip, attached to the Course Overview lesson) together with sales_transactions_raw.csv. Run it in Google Colab or Jupyter from top to bottom and complete the Your turn exercises.