Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Data Cleaning and Preprocessing with Python

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 3 — Prepare, Report and Reproduce

Period 11 · Scaling and Encoding

Print / Save as PDF
Lecture video — Period 11: Scaling and Encoding

Min–max and z-score scaling, one-hot and ordinal encoding, and why scaling parameters are calculated from training data only.

Lesson overview

In this period, we will prepare numeric and categorical data for analysis. We will study two scaling methods, Min–Max and Z-score, and two ways to encode categories. We will use the same small sales example throughout the lesson. By the end, you should be able to explain what each method changes and choose a suitable method. Keep the original columns because business reports still need meaningful units and labels.

Study notes

By the end of this lesson you can:

  • Apply Min-Max scaling and Z-score standardization to a numeric column, keeping the original.
  • Handle a constant column and state the standard deviation convention you use.
  • Encode a nominal category with one-hot indicators using pd.get_dummies.
  • Encode an ordered category with an explicit ordinal mapping.
  • Verify transformed columns and explain how to reuse parameters on new data.

Key idea: Scaling and encoding change the representation of values, not their business meaning. Keep the original columns, choose a method for the next analytical step, and record the rules.

The example

The lecture uses four transactions. quantity is 2, 4, 6 and 8. region contains North, South and Central, which have no natural ranking. A service level (Low, Medium, High) is added for teaching ordinal encoding; the business defines this order. Labels stay in English in the code. Earlier cleaning has already handled missing values, duplicates and invalid values.

Min-Max scaling

Min-Max expresses a value relative to the observed minimum and maximum:

scaled = (x - min) / (max - min)

Here min = 2, max = 8 and the range (span) = 6. A quantity of 4 becomes 2 / 6, about 0.33; the minimum becomes 0 and the maximum 1. 0 means the smallest observed quantity, not zero items sold. The order of values is preserved when the range is positive. The code stores the result in a new column quantity_mm and uses a conditional for a constant column, where the span would be 0; the lecture chooses 0 for that case and documents it. Scaling cannot repair a wrong price or an invalid quantity.

Z-score standardization

A Z-score measures distance from the mean in standard deviation units:

z = (x - mean) / std

The mean is 5. Using the population convention (ddof=0), the variance is 5 and the standard deviation is the square root of 5, about 2.24. A quantity of 8 has z about 1.34; a quantity of 2 has z about -1.34. Negative scores are valid; they do not mean negative sales. Set ddof=0 explicitly, because pandas uses ddof=1 by default. The result has mean close to 0 and standard deviation close to 1 (small differences come from floating-point arithmetic). Standardization does not remove skewness or fix outliers.

Choosing a scaling method

  • Min-Max: useful when you want a relative position on a fixed 0-1 reference.
  • Z-score: useful when you want values expressed around the mean in standard deviation units.
  • Both react to extreme values, so neither replaces the Period 10 outlier review.
  • Scaling helps distance-based calculations, but business totals need the original quantities.
  • A new value outside the fitted range can give a Min-Max result below 0 or above 1.

Encoding categories

If North = 1, South = 2, Central = 3, some methods would treat Central as "larger" than North, an unsupported relationship. One-hot encoding creates a separate 0/1 column for each category instead. With exactly one known region per row and all categories kept, each row's indicators add up to 1. pd.get_dummies(df["region"], prefix="region", dtype=int) does this; pd.concat joins the indicators to the table and aligns rows by index, so keep the index consistent. A missing label can produce an all-zero row unless you set a policy, and spelling variants create separate columns unless cleaned first.

Ordinal encoding suits categories with a meaningful order. Map them explicitly, {"Low": 0, "Medium": 1, "High": 2}, rather than relying on alphabetical order. A one-step difference in codes does not mean equal real differences between levels, and map returns missing for an unknown label, so check for that.

Worked example

The service levels follow the lecture (Low, Medium, High, Medium); the region order is illustrative.

import pandas as pd

df = pd.DataFrame({
    "transaction_id": ["T1", "T2", "T3", "T4"],
    "quantity": [2, 4, 6, 8],
    "region": ["North", "South", "Central", "North"],
    "service_level": ["Low", "Medium", "High", "Medium"],
})

x = df["quantity"]
span = x.max() - x.min()
df["quantity_mm"] = 0.0 if span == 0 else (x - x.min()) / span
df["quantity_z"] = (x - x.mean()) / x.std(ddof=0)

dummies = pd.get_dummies(df["region"], prefix="region", dtype=int)
assert (dummies.sum(axis=1) == 1).all()
LEVEL_ORDER = {"Low": 0, "Medium": 1, "High": 2}
df["service_code"] = df["service_level"].map(LEVEL_ORDER)
assert df["service_code"].notna().all()

out = pd.concat([df, dummies], axis=1)
print(out.round(2))
print(round((6 - x.min()) / span, 2), round((6 - x.mean()) / x.std(ddof=0), 2))

Expected output: quantity_mm = 0.0, 0.33, 0.67, 1.0; quantity_z = -1.34, -0.45, 0.45, 1.34; three region columns with one 1 per row; service codes 0, 1, 2, 1; and the last line prints 0.67 0.45, the lecture's check for a quantity of 6. The original quantity and region columns are unchanged, and there are still four rows.

In a prediction project, compute min, max, mean and standard deviation only from the training data, then reuse them for test data. Save the category list too, and decide how to handle a category that appears later.

Common mistakes

  • Overwriting the original column. Reports and revenue need real units; create quantity_mm and quantity_z instead.
  • Forgetting ddof=0. pandas' default std() uses ddof=1, so your Z-scores will not match the formula you state.
  • Numbering nominal categories. Codes 1, 2, 3 for regions invent an order; use one-hot encoding.
  • Fitting scaling on all data in a prediction task. Parameters must come from the training set only.

Summary

  • Min-Max: (x - min) / (max - min); 0 and 1 are the observed minimum and maximum.
  • Z-score: (x - mean) / std with ddof=0; negative scores are normal.
  • Handle constant columns explicitly and document the choice.
  • One-hot for categories without order; explicit ordinal mapping for business-defined order.
  • Verify row counts, unchanged originals, one-hot row sums and summary statistics after transforming.

Flashcards · Scaling and Encoding

9 words
WordMeaningExample
  • Min-Max scaling

    Rescales values to 0-1 using the observed minimum and maximum: (x - min) / (max - min).

    Quantity 4 with min 2, max 8 gives 0.33.

  • Z-score

    Distance from the mean in standard deviation units: (x - mean) / std.

    Quantity 8, mean 5, std 2.24 gives 1.34.

  • Constant column

    A column where every value is equal, so the Min-Max range is zero and needs a defined rule.

    `0.0 if span == 0 else (x - x.min()) / span`

  • ddof=0

    Population convention for standard deviation, dividing by n instead of n - 1.

    `x.std(ddof=0)` gives about 2.24 for 2, 4, 6, 8.

  • One-hot encoding

    Represents each category of a nominal variable as its own 0/1 indicator column.

    North row: region_North 1, region_South 0, region_Central 0.

  • pd.get_dummies()

    The pandas function that creates one-hot indicator columns from a categorical column.

    `pd.get_dummies(df["region"], prefix="region", dtype=int)`

  • Ordinal encoding

    Maps ordered categories to integers following a business-defined order.

    `{"Low": 0, "Medium": 1, "High": 2}`

  • Nominal variable

    A category with no natural order, such as a region.

    North, South, Central

  • Training-set parameters

    Scaling values (min, max, mean, std) computed only on training data and reused on test data.

    Fit mean and std on training rows, apply to test rows.