Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 3: Model Evaluation and Optimization

Applying the Complete Machine Learning Workflow

Print / Save as PDF

Applying the Complete Machine Learning Workflow

This lesson ties together everything you have learned into a single workflow.

Nine Steps

#StepQuestion to answer
1Define the problemWhat are we predicting? Regression or classification?
2Collect dataIs there enough? Does it represent reality?
3Explore (EDA)What is the distribution? How much is missing? Are there outliers?
4Split train/testDo we need to stratify?
5PreprocessImpute, normalize, encode — fit on train only
6BaselineA simple model as a point of comparison
7Train and tuneCross-validation on the training set
8Final evaluationRun the test set once
9Interpret and deployHow does the model go wrong? Is that acceptable?

Complete Skeleton Code

import pandas as pd
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix

data = pd.read_csv("data.csv")
target = "Outcome"
x, y = data.drop(target, axis=1), data[target]

x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.2, random_state=1009, stratify=y)

pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler",  StandardScaler()),
    ("model",   RandomForestClassifier(random_state=100)),
])

grid = {"model__max_depth": [5, 10, None], "model__n_estimators": [100, 300]}
search = GridSearchCV(pipe, grid, cv=5, scoring="f1", n_jobs=-1)
search.fit(x_train, y_train)

y_pred = search.best_estimator_.predict(x_test)   # the test set is run ONCE
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))

Step 9 Is the Step Most Often Skipped

Getting a classification_report does not mean you are done. Keep asking:

  • Which kind of error does the model make more often — FP or FN? Is that kind acceptable in the real-world context?
  • On which group does it go wrong? A model that is 85% accurate overall but only 60% accurate on a specific user group is a fairness problem, not a technical one.
  • Does the real data at deployment look like the training data?

The Most Common Mistake

Stopping at the accuracy number. A model is used by real people to make real decisions. Understanding where it goes wrong matters as much as how often it is right.

Study notes

By the end of this lesson you can:

  • List the nine steps of a machine learning workflow in the correct order
  • Explain why preprocessing is fitted on the training set only and why splitting comes first
  • Read and adapt a complete skeleton that uses Pipeline, GridSearchCV and a single test-set evaluation
  • Go beyond the metrics in step 9: analyse error types, groups and deployment data

Key idea: A trustworthy model comes from a disciplined order of steps (split before preprocessing, tune on training data only, test once) followed by a serious look at how and for whom the model goes wrong.

The nine steps

#StepQuestion to answer
1Define the problemWhat are we predicting? Regression or classification?
2Collect dataIs there enough? Does it represent reality?
3Explore (EDA)What is the distribution? How much is missing? Are there outliers?
4Split train/testDo we need to stratify?
5PreprocessImpute, normalise, encode, fitted on train only
6BaselineA simple model as a point of comparison
7Train and tuneCross-validation on the training set
8Final evaluationRun the test set once
9Interpret and deployHow does the model go wrong? Is that acceptable?

Two ordering rules carry most of the weight:

  • Split (step 4) before preprocessing (step 5). If you impute or scale on the full table, statistics from the test rows leak into training and the final score becomes optimistic.
  • Tune (step 7) before the final evaluation (step 8). The test set is used once, after all decisions have been made.

The complete skeleton

import pandas as pd
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix

data = pd.read_csv("data.csv")
target = "Outcome"
x, y = data.drop(target, axis=1), data[target]

x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.2, random_state=1009, stratify=y)

pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler",  StandardScaler()),
    ("model",   RandomForestClassifier(random_state=100)),
])

grid = {"model__max_depth": [5, 10, None], "model__n_estimators": [100, 300]}
search = GridSearchCV(pipe, grid, cv=5, scoring="f1", n_jobs=-1)
search.fit(x_train, y_train)

y_pred = search.best_estimator_.predict(x_test)   # the test set is run ONCE
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))

How the pieces map to the steps:

  • stratify=y keeps the class proportions equal in train and test (step 4).
  • The Pipeline bundles imputation (median), scaling and the model, so preprocessing is always fitted on training data, including inside every cross-validation fold (step 5).
  • GridSearchCV tunes max_depth and n_estimators with 5-fold cross-validation on x_train only, scored by F1 (step 7).
  • The test set appears only in the last three lines (step 8).

Step 9 is the step most often skipped

A classification_report does not mean you are done. Keep asking:

  • Which kind of error does the model make more often, FP or FN? Is that kind acceptable in the real context?
  • On which group does it go wrong? A model that is 85% accurate overall but only 60% accurate for a particular user group has a fairness problem, not just a technical one.
  • Does the real data at deployment look like the training data? If not, performance in use may differ from the test score.

Worked example

Suppose the skeleton above, run on a diabetes dataset with Outcome (1 = diabetes), gives a confusion matrix for the positive class with 30 true positives and 24 false negatives, and an overall accuracy of 0.77. Work through step 9:

  1. Error type. Recall for class 1 = 30 / (30 + 24) = 0.56. Almost half of the patients with diabetes are missed. For a screening tool, false negatives are the dangerous error, so this is not yet acceptable even though accuracy looks reasonable.
  2. Groups. Check accuracy by age group:
results = x_test.assign(actual=y_test, pred=y_pred)
results["age_group"] = pd.cut(results["Age"], bins=[0, 30, 50, 100])
results["correct"] = results["actual"] == results["pred"]
print(results.groupby("age_group", observed=True)["correct"].mean())

If one age group scores much lower than the others, investigate before deployment. 3. Deployment data. If the clinic's real patients are much older than the training data, the test score may not hold. 4. Action. Go back to step 7, for example tune with scoring="recall", or adjust the decision threshold, then evaluate on the test set once more only when the new approach is fixed.

Common mistakes

  • Preprocessing before splitting. Statistics from the test rows leak into training.
  • Using the test set inside the tuning loop. The final score is no longer honest.
  • Skipping the baseline. Without step 6 you cannot tell whether the tuned model is worth its complexity.
  • Stopping at the accuracy number. Real people make real decisions with the model; where it goes wrong matters as much as how often it is right.

Summary

  • Follow nine steps: define, collect, explore, split, preprocess, baseline, tune, evaluate once, interpret.
  • Split before preprocessing, and fit every preprocessing step on the training data only.
  • A Pipeline plus GridSearchCV keeps preprocessing and tuning inside the training set.
  • Run the test set exactly once with the final model.
  • Step 9: check error types, performance per group, and whether deployment data matches training data.

Flashcards · Applying the Complete Machine Learning Workflow

9 words
WordMeaningExample
  • Machine learning workflow

    The ordered steps from defining a problem to interpreting and deploying an evaluated model.

    Define, collect, EDA, split, preprocess, baseline, tune, test, interpret.

  • EDA (exploratory data analysis)

    Exploring the data's distributions, missing values and outliers before modelling.

    data.describe(); data.isna().sum()

  • stratify=y

    train_test_split option that keeps the class proportions the same in train and test sets.

    train_test_split(x, y, test_size=0.2, stratify=y)

  • SimpleImputer

    scikit-learn transformer that fills missing values, for example with the column median.

    SimpleImputer(strategy='median')

  • best_estimator_

    The model from GridSearchCV refitted with the best hyperparameters on the whole training set.

    search.best_estimator_.predict(x_test)

  • Single test evaluation

    Running the test set exactly once, after all modelling decisions are final.

    The last step before interpretation.

  • Error analysis

    Studying which kinds of errors a model makes and whether they are acceptable in context.

    24 false negatives in diabetes screening is serious.

  • Fairness problem

    A model that performs much worse for a specific group of people than overall.

    85% accuracy overall, 60% for one user group.

  • Data drift

    A difference between the data seen at deployment and the data used for training.

    Training patients aged 20-50, real patients mostly over 60.