Capstone Practice: Choosing Evaluation Metrics for Three Scenarios
[markdown] content available only on the platform
[markdown] content available only on the platform
Flashcards · Capstone Practice: Choosing Evaluation Metrics for Three Scenarios
Cost of errors — The real-world consequence of a false positive or a false negative, which drives the choice of metric.
A missed disease case costs far more than a false alarm.
Recall-first scenario — A situation where missing a positive case is the most expensive error.
Cancer screening.
Precision-first scenario — A situation where a false alarm is the most expensive error.
A spam filter blocking a contract email.
F1-first scenario — A situation where both false alarms and misses are costly, so a balanced metric is needed.
Fraud detection.
Misleading accuracy — High accuracy on imbalanced data that hides a model's failure on the minority class.
96% accuracy, 0 returns flagged.
MAE for stakeholders — MAE states the average error in the target's own unit, which non-technical people understand.
Off by 0.3 billion VND on average.
R-squared for model value — R-squared shows how much better the model is than always predicting the mean.
R-squared 0.8 vs the mean baseline at 0.
zero_division — classification_report argument that sets the value shown when a metric divides by zero, avoiding warnings.
classification_report(y, y_pred, zero_division=0)
