You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
Models only consume numbers. This lesson covers two jobs: bringing numerical columns onto the same scale, and turning text columns into numbers.
Suppose we predict house prices from two features:
| Feature | Value range |
|---|---|
| Area (m²) | 30 – 200 |
| Number of bedrooms | 1 – 5 |
With distance-based algorithms (KNN, SVM) or with gradient descent, the area column will dominate the bedroom column simply because its numbers are larger — not because it is more important.
Normalization (Min-Max) — squeeze into the range [0, 1]:
x_new = (x − x_min) / (x_max − x_min)
Standardization (z-score) — bring to mean 0, standard deviation 1:
x_new = (x − mean) / std
| Min-Max | z-score | |
|---|---|---|
| Output range | Fixed [0, 1] | Unbounded |
| Sensitive to outliers | Very sensitive — one extremely large value squashes everything else close to 0 | Less sensitive |
| Suitable when | The upper/lower bounds are well known, data is clean | There are outliers, distribution is close to normal |
from sklearn.preprocessing import StandardScaler, MinMaxScaler
scaler = StandardScaler()
x_train = scaler.fit_transform(x_train) # learn mean/std FROM train
x_test = scaler.transform(x_test) # apply that same mean/std to test
One-hot encoding for nominal variables (no order) — each category becomes a 0/1 column:
data = pd.get_dummies(data, columns=["city"])
# city → city_Hanoi, city_HCM, city_Hue
Ordinal encoding for ordered variables — preserves the greater/less relationship:
size_map = {"S": 0, "M": 1, "L": 2}
data["size"] = data["size"].map(size_map)
Assigning numbers to a nominal variable. If you encode Hanoi = 1, Ho Chi Minh City = 2, Hue = 3, the model will infer that Hue > Ho Chi Minh City > Hanoi and that the distance Hanoi–Hue is twice the distance Hanoi–Ho Chi Minh City. That is a completely fabricated relationship. For an unordered variable, you must use one-hot.
By the end of this lesson you can:
Key idea: Models only consume numbers, and those numbers must be comparable. Scale numerical columns with statistics learned from the training set, use one-hot encoding for unordered categories and ordinal codes only for ordered ones.
Suppose we predict house prices from two features:
| Feature | Value range |
|---|---|
| Area (m²) | 30 to 200 |
| Number of bedrooms | 1 to 5 |
Distance-based algorithms such as KNN and SVM, and models trained with gradient descent, will let the area column dominate simply because its numbers are larger, not because it matters more. A difference of 50 m² looks enormous next to a difference of 2 bedrooms, even if both are equally important for price.
Normalization (Min-Max) squeezes values into [0, 1]: x_new = (x - x_min) / (x_max - x_min).
Standardization (z-score) gives mean 0 and standard deviation 1: x_new = (x - mean) / std.
| Min-Max | z-score | |
|---|---|---|
| Output range | Fixed [0, 1] | Unbounded |
| Sensitivity to outliers | Very sensitive: one huge value squashes all others close to 0 | Less sensitive |
| Suitable when | Bounds are well known and data is clean | There are outliers, or the distribution is close to normal |
In scikit-learn these are MinMaxScaler and StandardScaler. The rule is the same for both: call fit_transform on the training set, so the scaler learns its statistics from train, and only transform on the test set, so test data is scaled with those same statistics.
One-hot encoding is for nominal variables with no order. Each category becomes its own 0/1 column; pd.get_dummies(data, columns=["city"]) turns city into city_Hanoi, city_HCM, city_Hue.
Ordinal encoding is for ordered variables and keeps the greater/less relationship. A simple way is a mapping: {"S": 0, "M": 1, "L": 2} applied with .map().
Assigning numbers to a nominal variable. If you encode Hanoi = 1, Ho Chi Minh City = 2, Hue = 3, the model concludes that Hue > Ho Chi Minh City > Hanoi and that the "distance" from Hanoi to Hue is twice the distance from Hanoi to Ho Chi Minh City. That relationship is completely invented. Unordered variables need one-hot encoding.
Scale area and bedrooms with statistics from the training set, then encode size and city.
import pandas as pd
from sklearn.preprocessing import StandardScaler
train = pd.DataFrame({"area": [30.0, 60.0, 90.0, 120.0], "bedrooms": [1.0, 2.0, 3.0, 4.0],
"city": ["Hanoi", "Hue", "HCM", "Hanoi"], "size": ["S", "M", "L", "M"]})
test = pd.DataFrame({"area": [200.0], "bedrooms": [5.0], "city": ["Hue"], "size": ["L"]})
num_cols = ["area", "bedrooms"]
scaler = StandardScaler()
train[num_cols] = scaler.fit_transform(train[num_cols]) # learn mean/std from train
test[num_cols] = scaler.transform(test[num_cols]) # reuse them on test
size_map = {"S": 0, "M": 1, "L": 2}
train["size"] = train["size"].map(size_map)
test["size"] = test["size"].map(size_map)
train = pd.get_dummies(train, columns=["city"], dtype=int)
test = pd.get_dummies(test, columns=["city"], dtype=int).reindex(columns=train.columns, fill_value=0)
print(train.round(3))
print(test.round(3))
Expected output:
area bedrooms size city_HCM city_Hanoi city_Hue
0 -1.342 -1.342 0 0 1 0
1 -0.447 -0.447 1 0 0 1
2 0.447 0.447 2 1 0 0
3 1.342 1.342 1 0 1 0
area bedrooms size city_HCM city_Hanoi city_Hue
0 3.727 2.236 2 0 0 1
After scaling, area and bedrooms sit on the same scale even though their raw ranges were very different. The test house is scaled with the training mean and standard deviation, so its large area shows up as 3.727 standard deviations above the training mean. reindex makes the test table have exactly the same columns as the training table, filling the missing city columns with 0.
get_dummies separately on train and test without aligning columns. The test table can end up with fewer or different columns.Flashcards · Feature Scaling & Encoding
8 wordsA feature with large raw values outweighing others in distance or gradient calculations just because of its scale.
Rescales a column into [0, 1] using its minimum and maximum; very sensitive to outliers.
Rescales a column to mean 0 and standard deviation 1; output is unbounded and less outlier-sensitive.
Learns the scaling statistics from the data and transforms it in one step; use it on the training set.
Applies statistics already learned from train to new data, such as the test set.
pandas function that one-hot encodes chosen columns into 0/1 columns per category.
Replaces each value using a dictionary; a simple way to ordinal-encode ordered categories.
An algorithm that compares examples by distance, so it is strongly affected by feature scale.