You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
This lab brings together the material from units 3.1 to 3.5: accuracy and imbalanced data, the confusion matrix, precision / recall / F1, and the regression metrics MAE and R². Do it after you have completed all five units.
The task is in the Assignment section below.
By the end of this lesson you can:
Key idea: The right metric follows from the cost of mistakes: decide what kind of error hurts most in the real situation, then pick the metric that exposes that error.
This capstone combines units 3.1 to 3.5: accuracy and imbalanced data, the confusion matrix, precision, recall and F1, and the regression metrics MAE and R-squared. The three scenarios and the task are in the Assignment section. These notes give a step-by-step method for answering any scenario.
Look at what the model outputs.
| Output | Metric family |
|---|---|
| a category (fraud / normal, sick / healthy, spam / not spam) | accuracy, precision, recall, F1 |
| a number (price, demand, time) | MAE, (R)MSE, R-squared |
If one class makes up, say, 95% or more of the data, accuracy is misleading: always predicting the majority class already scores 95%. In that case rely on per-class metrics for the minority class.
Use the confusion matrix vocabulary from unit 3.2:
A good answer explains what goes wrong with a poor metric. For example: "If we used accuracy for disease screening, a model that never detects disease would look 99% accurate and would be deployed, and every sick patient would be missed."
Scenario (practice, not one of the assignment scenarios): an online shop builds a model that flags orders likely to be returned, so staff can call the customer before shipping. Only 4% of orders are returned. A call costs a little staff time; a missed return costs shipping both ways.
Reasoning step by step:
from sklearn.metrics import classification_report, recall_score
y_true = [0] * 96 + [1] * 4
lazy = [0] * 100 # never flags a return
print(recall_score(y_true, lazy))
print(classification_report(y_true, lazy, zero_division=0))
Expected output: recall 0.0 for the lazy model, and a report showing accuracy 0.96 but precision, recall and F1 of 0.00 for class 1, a clear warning that accuracy is hiding the failure.
Flashcards · Capstone Practice: Choosing Evaluation Metrics for Three Scenarios
8 wordsThe real-world consequence of a false positive or a false negative, which drives the choice of metric.
A situation where missing a positive case is the most expensive error.
A situation where a false alarm is the most expensive error.
A situation where both false alarms and misses are costly, so a balanced metric is needed.
High accuracy on imbalanced data that hides a model's failure on the minority class.
MAE states the average error in the target's own unit, which non-technical people understand.
R-squared shows how much better the model is than always predicting the mean.
classification_report argument that sets the value shown when a metric divides by zero, avoiding warnings.