Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1: Data Preparation and Model Foundations

Feature Scaling & Encoding

Print / Save as PDF

Feature Scaling and Encoding Categorical Variables

Models only consume numbers. This lesson covers two jobs: bringing numerical columns onto the same scale, and turning text columns into numbers.

Why scaling is necessary

Suppose we predict house prices from two features:

FeatureValue range
Area (m²)30 – 200
Number of bedrooms1 – 5

With distance-based algorithms (KNN, SVM) or with gradient descent, the area column will dominate the bedroom column simply because its numbers are larger — not because it is more important.

Two scaling methods

Normalization (Min-Max) — squeeze into the range [0, 1]:

x_new = (x − x_min) / (x_max − x_min)

Standardization (z-score) — bring to mean 0, standard deviation 1:

x_new = (x − mean) / std

Min-Maxz-score
Output rangeFixed [0, 1]Unbounded
Sensitive to outliersVery sensitive — one extremely large value squashes everything else close to 0Less sensitive
Suitable whenThe upper/lower bounds are well known, data is cleanThere are outliers, distribution is close to normal
from sklearn.preprocessing import StandardScaler, MinMaxScaler
scaler = StandardScaler()
x_train = scaler.fit_transform(x_train)   # learn mean/std FROM train
x_test  = scaler.transform(x_test)        # apply that same mean/std to test

Encoding categorical variables

One-hot encoding for nominal variables (no order) — each category becomes a 0/1 column:

data = pd.get_dummies(data, columns=["city"])
# city → city_Hanoi, city_HCM, city_Hue

Ordinal encoding for ordered variables — preserves the greater/less relationship:

size_map = {"S": 0, "M": 1, "L": 2}
data["size"] = data["size"].map(size_map)

The most common mistake

Assigning numbers to a nominal variable. If you encode Hanoi = 1, Ho Chi Minh City = 2, Hue = 3, the model will infer that Hue > Ho Chi Minh City > Hanoi and that the distance Hanoi–Hue is twice the distance Hanoi–Ho Chi Minh City. That is a completely fabricated relationship. For an unordered variable, you must use one-hot.

Study notes

By the end of this lesson you can:

  • Explain why features on different scales distort distance-based algorithms and gradient descent.
  • Compare Min-Max normalization and z-score standardization, including their sensitivity to outliers.
  • Fit a scaler on the training set and apply the same scaling to the test set.
  • Encode nominal variables with one-hot encoding and ordered variables with a mapping.

Key idea: Models only consume numbers, and those numbers must be comparable. Scale numerical columns with statistics learned from the training set, use one-hot encoding for unordered categories and ordinal codes only for ordered ones.

Why scaling is necessary

Suppose we predict house prices from two features:

FeatureValue range
Area (m²)30 to 200
Number of bedrooms1 to 5

Distance-based algorithms such as KNN and SVM, and models trained with gradient descent, will let the area column dominate simply because its numbers are larger, not because it matters more. A difference of 50 m² looks enormous next to a difference of 2 bedrooms, even if both are equally important for price.

Two scaling methods

Normalization (Min-Max) squeezes values into [0, 1]: x_new = (x - x_min) / (x_max - x_min).

Standardization (z-score) gives mean 0 and standard deviation 1: x_new = (x - mean) / std.

Min-Maxz-score
Output rangeFixed [0, 1]Unbounded
Sensitivity to outliersVery sensitive: one huge value squashes all others close to 0Less sensitive
Suitable whenBounds are well known and data is cleanThere are outliers, or the distribution is close to normal

In scikit-learn these are MinMaxScaler and StandardScaler. The rule is the same for both: call fit_transform on the training set, so the scaler learns its statistics from train, and only transform on the test set, so test data is scaled with those same statistics.

Encoding categorical variables

One-hot encoding is for nominal variables with no order. Each category becomes its own 0/1 column; pd.get_dummies(data, columns=["city"]) turns city into city_Hanoi, city_HCM, city_Hue.

Ordinal encoding is for ordered variables and keeps the greater/less relationship. A simple way is a mapping: {"S": 0, "M": 1, "L": 2} applied with .map().

The most common mistake

Assigning numbers to a nominal variable. If you encode Hanoi = 1, Ho Chi Minh City = 2, Hue = 3, the model concludes that Hue > Ho Chi Minh City > Hanoi and that the "distance" from Hanoi to Hue is twice the distance from Hanoi to Ho Chi Minh City. That relationship is completely invented. Unordered variables need one-hot encoding.

Worked example

Scale area and bedrooms with statistics from the training set, then encode size and city.

import pandas as pd
from sklearn.preprocessing import StandardScaler

train = pd.DataFrame({"area": [30.0, 60.0, 90.0, 120.0], "bedrooms": [1.0, 2.0, 3.0, 4.0],
                      "city": ["Hanoi", "Hue", "HCM", "Hanoi"], "size": ["S", "M", "L", "M"]})
test = pd.DataFrame({"area": [200.0], "bedrooms": [5.0], "city": ["Hue"], "size": ["L"]})

num_cols = ["area", "bedrooms"]
scaler = StandardScaler()
train[num_cols] = scaler.fit_transform(train[num_cols])   # learn mean/std from train
test[num_cols] = scaler.transform(test[num_cols])         # reuse them on test

size_map = {"S": 0, "M": 1, "L": 2}
train["size"] = train["size"].map(size_map)
test["size"] = test["size"].map(size_map)

train = pd.get_dummies(train, columns=["city"], dtype=int)
test = pd.get_dummies(test, columns=["city"], dtype=int).reindex(columns=train.columns, fill_value=0)

print(train.round(3))
print(test.round(3))

Expected output:

    area  bedrooms  size  city_HCM  city_Hanoi  city_Hue
0 -1.342    -1.342     0         0           1         0
1 -0.447    -0.447     1         0           0         1
2  0.447     0.447     2         1           0         0
3  1.342     1.342     1         0           1         0
    area  bedrooms  size  city_HCM  city_Hanoi  city_Hue
0  3.727     2.236     2         0           0         1

After scaling, area and bedrooms sit on the same scale even though their raw ranges were very different. The test house is scaled with the training mean and standard deviation, so its large area shows up as 3.727 standard deviations above the training mean. reindex makes the test table have exactly the same columns as the training table, filling the missing city columns with 0.

Common mistakes

  • Fitting the scaler on the test set. Fit on train, transform test; otherwise test information leaks into preprocessing.
  • Using Min-Max on columns with extreme values. One CEO salary will squash every other salary towards 0; prefer z-score there.
  • Giving nominal categories ordered numbers. Use one-hot encoding for cities, colours and similar columns.
  • Calling get_dummies separately on train and test without aligning columns. The test table can end up with fewer or different columns.

Summary

  • Without scaling, large-valued features dominate distance-based models and gradient descent.
  • Min-Max gives [0, 1] but is very sensitive to outliers; z-score gives mean 0 and std 1 and is less sensitive.
  • Fit scalers on train only, then transform test with the same statistics.
  • Use one-hot encoding for nominal variables and ordinal codes for ordered ones.
  • Never invent an order for an unordered category.

Flashcards · Feature Scaling & Encoding

8 words
WordMeaningExample
  • Scale dominance

    A feature with large raw values outweighing others in distance or gradient calculations just because of its scale.

    Area 30-200 m² swamping bedrooms 1-5.

  • Min-Max normalization

    Rescales a column into [0, 1] using its minimum and maximum; very sensitive to outliers.

    MinMaxScaler().fit_transform(X_train)

  • z-score standardization

    Rescales a column to mean 0 and standard deviation 1; output is unbounded and less outlier-sensitive.

    StandardScaler().fit_transform(X_train)

  • scaler.fit_transform()

    Learns the scaling statistics from the data and transforms it in one step; use it on the training set.

    x_train = scaler.fit_transform(x_train)

  • scaler.transform()

    Applies statistics already learned from train to new data, such as the test set.

    x_test = scaler.transform(x_test)

  • pd.get_dummies()

    pandas function that one-hot encodes chosen columns into 0/1 columns per category.

    pd.get_dummies(data, columns=["city"], dtype=int)

  • Series.map()

    Replaces each value using a dictionary; a simple way to ordinal-encode ordered categories.

    data["size"].map({"S": 0, "M": 1, "L": 2})

  • Distance-based algorithm

    An algorithm that compares examples by distance, so it is strongly affected by feature scale.

    KNN, SVM