Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 3: Model Evaluation and Optimization

Hyperparameter Tuning

Print / Save as PDF

Hyperparameter Tuning

How Parameters and Hyperparameters Differ

ParameterHyperparameter
Who decidesThe model learns it from the dataThe user sets it before training
ExamplesThe coefficient w of linear regressionmax_depth, C, n_estimators

Grid Search

List the values you want to try; the library runs every combination and picks the best one according to cross-validation.

from sklearn.model_selection import GridSearchCV

grid = {
    "model__max_depth": [3, 5, 10, None],
    "model__n_estimators": [100, 300],
}
search = GridSearchCV(pipe, grid, cv=5, scoring="f1", n_jobs=-1)
search.fit(x_train, y_train)

print(search.best_params_)
print(search.best_score_)

Note the count: 4 × 2 = 8 combinations, and each combination runs 5 folds → 40 model trainings. The wider the grid, the cost grows exponentially.

Random Search — Often More Effective

Instead of trying everything, draw N random combinations. It sounds worse, but in practice it is often better for the same computational budget: most hyperparameters have little effect, so grid search spends a great many runs probing values that change nothing.

from sklearn.model_selection import RandomizedSearchCV
search = RandomizedSearchCV(pipe, grid, n_iter=20, cv=5, random_state=1009)

The Biggest Trap

If you tune based on the test set and then report the result on that same set, that number is no longer honest — you have chosen the model that fits the test set best. The correct workflow:

  1. Split train / test right from the start
  2. Tune with cross-validation on the training set only
  3. Run the test set exactly once, with the model you have settled on

The Most Common Mistake

Running a huge grid from the very beginning. Run a coarse search over a wide range first to find a good region, then run a fine search around that region — this saves hours of computation.

Study notes

By the end of this lesson you can:

  • Distinguish parameters (learned) from hyperparameters (set by you)
  • Run a grid search with GridSearchCV and count how many models it trains
  • Explain why random search is often more efficient than grid search
  • Follow the correct workflow so the test score stays honest
  • Plan a coarse-to-fine search instead of one huge grid

Key idea: Tune hyperparameters with cross-validation on the training set only, search coarse-to-fine, and touch the test set exactly once at the end.

Parameters versus hyperparameters

ParameterHyperparameter
Who decidesthe model learns it from the datathe user sets it before training
Examplesthe coefficient w of Linear Regressionmax_depth, C, n_estimators

Hyperparameters control how the model learns: how deep a tree may grow, how many trees a forest has, how wide an SVM margin is allowed to be. Different values can give very different results, so we search for good ones.

Grid search

List the values you want to try; the library trains every combination and picks the best according to cross-validation.

from sklearn.model_selection import GridSearchCV

# pipe is a Pipeline whose last step is named "model" (built in the worked example below)
grid = {
    "model__max_depth": [3, 5, 10, None],
    "model__n_estimators": [100, 300],
}
search = GridSearchCV(pipe, grid, cv=5, scoring="f1", n_jobs=-1)
search.fit(x_train, y_train)

print(search.best_params_)
print(search.best_score_)

Notes on the code:

  • The names model__max_depth use a double underscore: "the max_depth of the pipeline step called model".
  • n_jobs=-1 uses all CPU cores.
  • best_params_ is the best combination; best_score_ is its mean cross-validation score.

Count the cost: 4 x 2 = 8 combinations, each run on 5 folds, gives 40 model trainings. Every new hyperparameter multiplies the number of combinations, so the cost grows very quickly as the grid widens.

Random search: often more effective

Instead of trying every combination, draw N random combinations:

from sklearn.model_selection import RandomizedSearchCV
search = RandomizedSearchCV(pipe, grid, n_iter=20, cv=5, random_state=1009)

It sounds worse, but for the same computing budget it is often better. Most hyperparameters have little effect; grid search spends many runs probing values that change nothing, while random search tries more distinct values of the hyperparameters that do matter.

The biggest trap: tuning on the test set

If you choose hyperparameters by their test-set score and then report that same score, the number is no longer honest: you have picked the model that happens to fit the test set best. The correct workflow:

  1. split train and test right at the start;
  2. tune with cross-validation on the training set only;
  3. run the test set exactly once, with the model you have settled on.

Search coarse-to-fine

Running a huge grid from the start wastes hours. First run a coarse search over a wide range (for example max_depth in [2, 5, 10, 20, None]) to find a good region, then a fine search around it (for example [4, 5, 6, 7]).

Worked example

Plan a tuning run for a Random Forest pipeline and count its cost.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

pipe = Pipeline([("scaler", StandardScaler()),
                 ("model", RandomForestClassifier(random_state=100))])

grid = {"model__max_depth": [3, 5, 10],
        "model__n_estimators": [100, 200, 300]}
search = GridSearchCV(pipe, grid, cv=5, scoring="f1")
search.fit(x_train, y_train)

print(search.best_params_, round(search.best_score_, 3))
print(round(search.score(x_test, y_test), 3))   # first and only use of the test set

Step by step:

  1. Combinations: 3 x 3 = 9. With 5 folds that is 9 x 5 = 45 trainings, plus one final refit of the best combination on the whole training set.
  2. best_params_ shows the winning pair, for example {'model__max_depth': 10, 'model__n_estimators': 300}; your values depend on the data.
  3. Only after the search is finished is the test set used, once, to report an honest score.
  4. If the best depth is at the edge of the grid (10), run a finer search above it, such as [10, 15, 20].

Common mistakes

  • Running a huge grid from the very beginning. Search coarse first, then fine.
  • Choosing hyperparameters on the test set. The reported score becomes optimistic and dishonest.
  • Confusing parameters and hyperparameters. Coefficients are learned; max_depth or C are set by you.
  • Forgetting the step prefix in a pipeline grid. Inside a pipeline, write model__max_depth, not max_depth.

Summary

  • Parameters are learned from data; hyperparameters are set before training.
  • GridSearchCV tries every combination: trainings = combinations x folds (+1 refit).
  • RandomizedSearchCV samples N combinations and is often more efficient for the same budget.
  • Tune on the training set with cross-validation; use the test set once at the end.
  • Search coarse-to-fine to save computation.

Flashcards · Hyperparameter Tuning

9 words
WordMeaningExample
  • Parameter

    A value the model learns from the training data.

    The coefficient w in Linear Regression.

  • Hyperparameter

    A setting chosen by the user before training that controls how the model learns.

    max_depth, C, n_estimators

  • Grid search

    Trying every combination of listed hyperparameter values and keeping the best by cross-validation.

    GridSearchCV(pipe, grid, cv=5)

  • Number of trainings

    Grid search trains combinations x folds models (plus one refit).

    4 x 2 combinations x 5 folds = 40.

  • Random search

    Trying a fixed number of randomly drawn hyperparameter combinations; often more efficient than a grid.

    RandomizedSearchCV(pipe, grid, n_iter=20, cv=5)

  • best_params_

    The hyperparameter combination with the highest mean cross-validation score.

    {'model__max_depth': 5, 'model__n_estimators': 300}

  • Step prefix (double underscore)

    In a pipeline grid, a hyperparameter is named step name + __ + parameter.

    'model__max_depth'

  • Coarse-to-fine search

    Search a wide range with few values first, then search densely around the best region.

    [2, 5, 10, 20] then [4, 5, 6, 7].

  • Honest test score

    A test score from a model whose hyperparameters were chosen without looking at the test set.

    search.score(x_test, y_test) run once at the end.