You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
This lesson ties together everything you have learned into a single workflow.
| # | Step | Question to answer |
|---|---|---|
| 1 | Define the problem | What are we predicting? Regression or classification? |
| 2 | Collect data | Is there enough? Does it represent reality? |
| 3 | Explore (EDA) | What is the distribution? How much is missing? Are there outliers? |
| 4 | Split train/test | Do we need to stratify? |
| 5 | Preprocess | Impute, normalize, encode — fit on train only |
| 6 | Baseline | A simple model as a point of comparison |
| 7 | Train and tune | Cross-validation on the training set |
| 8 | Final evaluation | Run the test set once |
| 9 | Interpret and deploy | How does the model go wrong? Is that acceptable? |
import pandas as pd
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix
data = pd.read_csv("data.csv")
target = "Outcome"
x, y = data.drop(target, axis=1), data[target]
x_train, x_test, y_train, y_test = train_test_split(
x, y, test_size=0.2, random_state=1009, stratify=y)
pipe = Pipeline([
("imputer", SimpleImputer(strategy="median")),
("scaler", StandardScaler()),
("model", RandomForestClassifier(random_state=100)),
])
grid = {"model__max_depth": [5, 10, None], "model__n_estimators": [100, 300]}
search = GridSearchCV(pipe, grid, cv=5, scoring="f1", n_jobs=-1)
search.fit(x_train, y_train)
y_pred = search.best_estimator_.predict(x_test) # the test set is run ONCE
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
Getting a classification_report does not mean you are done. Keep asking:
Stopping at the accuracy number. A model is used by real people to make real decisions. Understanding where it goes wrong matters as much as how often it is right.
By the end of this lesson you can:
Pipeline, GridSearchCV and a single test-set evaluationKey idea: A trustworthy model comes from a disciplined order of steps (split before preprocessing, tune on training data only, test once) followed by a serious look at how and for whom the model goes wrong.
| # | Step | Question to answer |
|---|---|---|
| 1 | Define the problem | What are we predicting? Regression or classification? |
| 2 | Collect data | Is there enough? Does it represent reality? |
| 3 | Explore (EDA) | What is the distribution? How much is missing? Are there outliers? |
| 4 | Split train/test | Do we need to stratify? |
| 5 | Preprocess | Impute, normalise, encode, fitted on train only |
| 6 | Baseline | A simple model as a point of comparison |
| 7 | Train and tune | Cross-validation on the training set |
| 8 | Final evaluation | Run the test set once |
| 9 | Interpret and deploy | How does the model go wrong? Is that acceptable? |
Two ordering rules carry most of the weight:
import pandas as pd
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix
data = pd.read_csv("data.csv")
target = "Outcome"
x, y = data.drop(target, axis=1), data[target]
x_train, x_test, y_train, y_test = train_test_split(
x, y, test_size=0.2, random_state=1009, stratify=y)
pipe = Pipeline([
("imputer", SimpleImputer(strategy="median")),
("scaler", StandardScaler()),
("model", RandomForestClassifier(random_state=100)),
])
grid = {"model__max_depth": [5, 10, None], "model__n_estimators": [100, 300]}
search = GridSearchCV(pipe, grid, cv=5, scoring="f1", n_jobs=-1)
search.fit(x_train, y_train)
y_pred = search.best_estimator_.predict(x_test) # the test set is run ONCE
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
How the pieces map to the steps:
stratify=y keeps the class proportions equal in train and test (step 4).Pipeline bundles imputation (median), scaling and the model, so preprocessing is always fitted on training data, including inside every cross-validation fold (step 5).GridSearchCV tunes max_depth and n_estimators with 5-fold cross-validation on x_train only, scored by F1 (step 7).A classification_report does not mean you are done. Keep asking:
Suppose the skeleton above, run on a diabetes dataset with Outcome (1 = diabetes), gives a confusion matrix for the positive class with 30 true positives and 24 false negatives, and an overall accuracy of 0.77. Work through step 9:
results = x_test.assign(actual=y_test, pred=y_pred)
results["age_group"] = pd.cut(results["Age"], bins=[0, 30, 50, 100])
results["correct"] = results["actual"] == results["pred"]
print(results.groupby("age_group", observed=True)["correct"].mean())
If one age group scores much lower than the others, investigate before deployment.
3. Deployment data. If the clinic's real patients are much older than the training data, the test score may not hold.
4. Action. Go back to step 7, for example tune with scoring="recall", or adjust the decision threshold, then evaluate on the test set once more only when the new approach is fixed.
Pipeline plus GridSearchCV keeps preprocessing and tuning inside the training set.Flashcards · Applying the Complete Machine Learning Workflow
9 wordsThe ordered steps from defining a problem to interpreting and deploying an evaluated model.
Exploring the data's distributions, missing values and outliers before modelling.
train_test_split option that keeps the class proportions the same in train and test sets.
scikit-learn transformer that fills missing values, for example with the column median.
The model from GridSearchCV refitted with the best hyperparameters on the whole training set.
Running the test set exactly once, after all modelling decisions are final.
Studying which kinds of errors a model makes and whether they are acceptable in context.
A model that performs much worse for a specific group of people than overall.
A difference between the data seen at deployment and the data used for training.