You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
| Parameter | Hyperparameter | |
|---|---|---|
| Who decides | The model learns it from the data | The user sets it before training |
| Examples | The coefficient w of linear regression | max_depth, C, n_estimators |
List the values you want to try; the library runs every combination and picks the best one according to cross-validation.
from sklearn.model_selection import GridSearchCV
grid = {
"model__max_depth": [3, 5, 10, None],
"model__n_estimators": [100, 300],
}
search = GridSearchCV(pipe, grid, cv=5, scoring="f1", n_jobs=-1)
search.fit(x_train, y_train)
print(search.best_params_)
print(search.best_score_)
Note the count: 4 × 2 = 8 combinations, and each combination runs 5 folds → 40 model trainings. The wider the grid, the cost grows exponentially.
Instead of trying everything, draw N random combinations. It sounds worse, but in practice it is often better for the same computational budget: most hyperparameters have little effect, so grid search spends a great many runs probing values that change nothing.
from sklearn.model_selection import RandomizedSearchCV
search = RandomizedSearchCV(pipe, grid, n_iter=20, cv=5, random_state=1009)
If you tune based on the test set and then report the result on that same set, that number is no longer honest — you have chosen the model that fits the test set best. The correct workflow:
Running a huge grid from the very beginning. Run a coarse search over a wide range first to find a good region, then run a fine search around that region — this saves hours of computation.
By the end of this lesson you can:
GridSearchCV and count how many models it trainsKey idea: Tune hyperparameters with cross-validation on the training set only, search coarse-to-fine, and touch the test set exactly once at the end.
| Parameter | Hyperparameter | |
|---|---|---|
| Who decides | the model learns it from the data | the user sets it before training |
| Examples | the coefficient w of Linear Regression | max_depth, C, n_estimators |
Hyperparameters control how the model learns: how deep a tree may grow, how many trees a forest has, how wide an SVM margin is allowed to be. Different values can give very different results, so we search for good ones.
List the values you want to try; the library trains every combination and picks the best according to cross-validation.
from sklearn.model_selection import GridSearchCV
# pipe is a Pipeline whose last step is named "model" (built in the worked example below)
grid = {
"model__max_depth": [3, 5, 10, None],
"model__n_estimators": [100, 300],
}
search = GridSearchCV(pipe, grid, cv=5, scoring="f1", n_jobs=-1)
search.fit(x_train, y_train)
print(search.best_params_)
print(search.best_score_)
Notes on the code:
model__max_depth use a double underscore: "the max_depth of the pipeline step called model".n_jobs=-1 uses all CPU cores.best_params_ is the best combination; best_score_ is its mean cross-validation score.Count the cost: 4 x 2 = 8 combinations, each run on 5 folds, gives 40 model trainings. Every new hyperparameter multiplies the number of combinations, so the cost grows very quickly as the grid widens.
Instead of trying every combination, draw N random combinations:
from sklearn.model_selection import RandomizedSearchCV
search = RandomizedSearchCV(pipe, grid, n_iter=20, cv=5, random_state=1009)
It sounds worse, but for the same computing budget it is often better. Most hyperparameters have little effect; grid search spends many runs probing values that change nothing, while random search tries more distinct values of the hyperparameters that do matter.
If you choose hyperparameters by their test-set score and then report that same score, the number is no longer honest: you have picked the model that happens to fit the test set best. The correct workflow:
Running a huge grid from the start wastes hours. First run a coarse search over a wide range (for example max_depth in [2, 5, 10, 20, None]) to find a good region, then a fine search around it (for example [4, 5, 6, 7]).
Plan a tuning run for a Random Forest pipeline and count its cost.
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
pipe = Pipeline([("scaler", StandardScaler()),
("model", RandomForestClassifier(random_state=100))])
grid = {"model__max_depth": [3, 5, 10],
"model__n_estimators": [100, 200, 300]}
search = GridSearchCV(pipe, grid, cv=5, scoring="f1")
search.fit(x_train, y_train)
print(search.best_params_, round(search.best_score_, 3))
print(round(search.score(x_test, y_test), 3)) # first and only use of the test set
Step by step:
best_params_ shows the winning pair, for example {'model__max_depth': 10, 'model__n_estimators': 300}; your values depend on the data.max_depth or C are set by you.model__max_depth, not max_depth.GridSearchCV tries every combination: trainings = combinations x folds (+1 refit).RandomizedSearchCV samples N combinations and is often more efficient for the same budget.Flashcards · Hyperparameter Tuning
9 wordsA value the model learns from the training data.
A setting chosen by the user before training that controls how the model learns.
Trying every combination of listed hyperparameter values and keeping the best by cross-validation.
Grid search trains combinations x folds models (plus one refit).
Trying a fixed number of randomly drawn hyperparameter combinations; often more efficient than a grid.
The hyperparameter combination with the highest mean cross-validation score.
In a pipeline grid, a hyperparameter is named step name + __ + parameter.
Search a wide range with few values first, then search densely around the best region.
A test score from a model whose hyperparameters were chosen without looking at the test set.