You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
In this period, we will prepare numeric and categorical data for analysis. We will study two scaling methods, Min–Max and Z-score, and two ways to encode categories. We will use the same small sales example throughout the lesson. By the end, you should be able to explain what each method changes and choose a suitable method. Keep the original columns because business reports still need meaningful units and labels.
By the end of this lesson you can:
pd.get_dummies.Key idea: Scaling and encoding change the representation of values, not their business meaning. Keep the original columns, choose a method for the next analytical step, and record the rules.
The lecture uses four transactions. quantity is 2, 4, 6 and 8. region contains North, South and Central, which have no natural ranking. A service level (Low, Medium, High) is added for teaching ordinal encoding; the business defines this order. Labels stay in English in the code. Earlier cleaning has already handled missing values, duplicates and invalid values.
Min-Max expresses a value relative to the observed minimum and maximum:
scaled = (x - min) / (max - min)
Here min = 2, max = 8 and the range (span) = 6. A quantity of 4 becomes 2 / 6, about 0.33; the minimum becomes 0 and the maximum 1. 0 means the smallest observed quantity, not zero items sold. The order of values is preserved when the range is positive. The code stores the result in a new column quantity_mm and uses a conditional for a constant column, where the span would be 0; the lecture chooses 0 for that case and documents it. Scaling cannot repair a wrong price or an invalid quantity.
A Z-score measures distance from the mean in standard deviation units:
z = (x - mean) / std
The mean is 5. Using the population convention (ddof=0), the variance is 5 and the standard deviation is the square root of 5, about 2.24. A quantity of 8 has z about 1.34; a quantity of 2 has z about -1.34. Negative scores are valid; they do not mean negative sales. Set ddof=0 explicitly, because pandas uses ddof=1 by default. The result has mean close to 0 and standard deviation close to 1 (small differences come from floating-point arithmetic). Standardization does not remove skewness or fix outliers.
If North = 1, South = 2, Central = 3, some methods would treat Central as "larger" than North, an unsupported relationship. One-hot encoding creates a separate 0/1 column for each category instead. With exactly one known region per row and all categories kept, each row's indicators add up to 1. pd.get_dummies(df["region"], prefix="region", dtype=int) does this; pd.concat joins the indicators to the table and aligns rows by index, so keep the index consistent. A missing label can produce an all-zero row unless you set a policy, and spelling variants create separate columns unless cleaned first.
Ordinal encoding suits categories with a meaningful order. Map them explicitly, {"Low": 0, "Medium": 1, "High": 2}, rather than relying on alphabetical order. A one-step difference in codes does not mean equal real differences between levels, and map returns missing for an unknown label, so check for that.
The service levels follow the lecture (Low, Medium, High, Medium); the region order is illustrative.
import pandas as pd
df = pd.DataFrame({
"transaction_id": ["T1", "T2", "T3", "T4"],
"quantity": [2, 4, 6, 8],
"region": ["North", "South", "Central", "North"],
"service_level": ["Low", "Medium", "High", "Medium"],
})
x = df["quantity"]
span = x.max() - x.min()
df["quantity_mm"] = 0.0 if span == 0 else (x - x.min()) / span
df["quantity_z"] = (x - x.mean()) / x.std(ddof=0)
dummies = pd.get_dummies(df["region"], prefix="region", dtype=int)
assert (dummies.sum(axis=1) == 1).all()
LEVEL_ORDER = {"Low": 0, "Medium": 1, "High": 2}
df["service_code"] = df["service_level"].map(LEVEL_ORDER)
assert df["service_code"].notna().all()
out = pd.concat([df, dummies], axis=1)
print(out.round(2))
print(round((6 - x.min()) / span, 2), round((6 - x.mean()) / x.std(ddof=0), 2))
Expected output: quantity_mm = 0.0, 0.33, 0.67, 1.0; quantity_z = -1.34, -0.45, 0.45, 1.34; three region columns with one 1 per row; service codes 0, 1, 2, 1; and the last line prints 0.67 0.45, the lecture's check for a quantity of 6. The original quantity and region columns are unchanged, and there are still four rows.
In a prediction project, compute min, max, mean and standard deviation only from the training data, then reuse them for test data. Save the category list too, and decide how to handle a category that appears later.
quantity_mm and quantity_z instead.ddof=0. pandas' default std() uses ddof=1, so your Z-scores will not match the formula you state.ddof=0; negative scores are normal.Flashcards · Scaling and Encoding
9 wordsRescales values to 0-1 using the observed minimum and maximum: (x - min) / (max - min).
Distance from the mean in standard deviation units: (x - mean) / std.
A column where every value is equal, so the Min-Max range is zero and needs a defined rule.
Population convention for standard deviation, dividing by n instead of n - 1.
Represents each category of a nominal variable as its own 0/1 indicator column.
The pandas function that creates one-hot indicator columns from a categorical column.
Maps ordered categories to integers following a business-defined order.
A category with no natural order, such as a region.
Scaling values (min, max, mean, std) computed only on training data and reused on test data.