Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 3: Model Evaluation and Optimization

Capstone Practice: Choosing Evaluation Metrics for Three Scenarios

Print / Save as PDF

Capstone Practice: Choosing Evaluation Metrics

This lab brings together the material from units 3.1 to 3.5: accuracy and imbalanced data, the confusion matrix, precision / recall / F1, and the regression metrics MAE and R². Do it after you have completed all five units.

The task is in the Assignment section below.

Study notes

By the end of this lesson you can:

  • Decide whether a scenario needs classification or regression metrics
  • Identify which error (false positive or false negative) is more expensive in a scenario
  • Choose between accuracy, precision, recall, F1, MAE and R-squared and justify the choice
  • Explain the real-world cost of choosing the wrong metric

Key idea: The right metric follows from the cost of mistakes: decide what kind of error hurts most in the real situation, then pick the metric that exposes that error.

What this practice asks you to do

This capstone combines units 3.1 to 3.5: accuracy and imbalanced data, the confusion matrix, precision, recall and F1, and the regression metrics MAE and R-squared. The three scenarios and the task are in the Assignment section. These notes give a step-by-step method for answering any scenario.

Step 1: classification or regression?

Look at what the model outputs.

OutputMetric family
a category (fraud / normal, sick / healthy, spam / not spam)accuracy, precision, recall, F1
a number (price, demand, time)MAE, (R)MSE, R-squared

Step 2 (classification): check the class balance

If one class makes up, say, 95% or more of the data, accuracy is misleading: always predicting the majority class already scores 95%. In that case rely on per-class metrics for the minority class.

Step 3 (classification): which error is more expensive?

Use the confusion matrix vocabulary from unit 3.2:

  • False negative (miss) is worse -> prioritise recall = TP / (TP + FN). Example: disease screening, where a missed patient goes untreated.
  • False positive (false alarm) is worse -> prioritise precision = TP / (TP + FP). Example: a spam filter, where a blocked important email can cost a contract.
  • Both are costly -> use F1, the harmonic mean, which stays low if either precision or recall is low. Example: fraud detection, where misses lose money and false alarms annoy customers.
  • Balanced classes and similar error costs -> accuracy can be acceptable, ideally reported with the other metrics.

Step 4 (regression): what do stakeholders need to know?

  • MAE answers "how far off are we on average?" in the target's unit. Easy to explain: "our price estimates are off by about 0.3 billion VND on average".
  • R-squared answers "how much better are we than always guessing the average?". Useful to show that the model adds value beyond a naive baseline.
  • If a few very large errors are especially dangerous, add RMSE, which penalises large errors more heavily than MAE.

Step 5: state the cost of the wrong choice

A good answer explains what goes wrong with a poor metric. For example: "If we used accuracy for disease screening, a model that never detects disease would look 99% accurate and would be deployed, and every sick patient would be missed."

Worked example

Scenario (practice, not one of the assignment scenarios): an online shop builds a model that flags orders likely to be returned, so staff can call the customer before shipping. Only 4% of orders are returned. A call costs a little staff time; a missed return costs shipping both ways.

Reasoning step by step:

  1. Output is "returned / not returned": classification.
  2. Only 4% returns: imbalanced, so accuracy is misleading (always "not returned" scores 96%).
  3. Error costs: a false positive wastes one phone call (cheap); a false negative costs two shipments (expensive). Misses are worse, so recall is the main metric, with precision monitored so staff are not flooded with calls.
  4. Wrong choice: optimising accuracy would reward a model that never flags anything.
from sklearn.metrics import classification_report, recall_score

y_true = [0] * 96 + [1] * 4
lazy = [0] * 100                      # never flags a return
print(recall_score(y_true, lazy))
print(classification_report(y_true, lazy, zero_division=0))

Expected output: recall 0.0 for the lazy model, and a report showing accuracy 0.96 but precision, recall and F1 of 0.00 for class 1, a clear warning that accuracy is hiding the failure.

Common mistakes

  • Defaulting to accuracy. On imbalanced data it rewards models that ignore the minority class.
  • Choosing a metric without naming the costly error. The justification must link the metric to FN or FP costs.
  • Using classification metrics for numeric targets. Prices and quantities need MAE, RMSE or R-squared.
  • Reporting R-squared alone to business users. Pair it with MAE so they see the error in real units.

Summary

  • First decide: classification metrics or regression metrics.
  • For classification, check class balance; accuracy fails on imbalanced data.
  • Costly misses -> recall; costly false alarms -> precision; both costly -> F1.
  • For regression, MAE gives the average error in real units, R-squared shows improvement over the mean, RMSE stresses large errors.
  • Always explain the real-world cost of choosing the wrong metric.

Flashcards · Capstone Practice: Choosing Evaluation Metrics for Three Scenarios

8 words
WordMeaningExample
  • Cost of errors

    The real-world consequence of a false positive or a false negative, which drives the choice of metric.

    A missed disease case costs far more than a false alarm.

  • Recall-first scenario

    A situation where missing a positive case is the most expensive error.

    Cancer screening.

  • Precision-first scenario

    A situation where a false alarm is the most expensive error.

    A spam filter blocking a contract email.

  • F1-first scenario

    A situation where both false alarms and misses are costly, so a balanced metric is needed.

    Fraud detection.

  • Misleading accuracy

    High accuracy on imbalanced data that hides a model's failure on the minority class.

    96% accuracy, 0 returns flagged.

  • MAE for stakeholders

    MAE states the average error in the target's own unit, which non-technical people understand.

    Off by 0.3 billion VND on average.

  • R-squared for model value

    R-squared shows how much better the model is than always predicting the mean.

    R-squared 0.8 vs the mean baseline at 0.

  • zero_division

    classification_report argument that sets the value shown when a metric divides by zero, avoiding warnings.

    classification_report(y, y_pred, zero_division=0)