Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 3: Model Evaluation and Optimization

Evaluation Metrics

Print / Save as PDF

Evaluation Metrics

The Confusion Matrix — the Root of All Classification Metrics

Predicted +Predicted −
Actual +TPFN (Type II error)
Actual −FP (Type I error)TN

Four Classification Metrics

MetricFormulaQuestion it answers
Accuracy(TP+TN)/totalWhat percentage did we get right?
PrecisionTP/(TP+FP)When the model says positive, how often is it right?
RecallTP/(TP+FN)Of the truly positive cases, how many did the model find?
F12PR/(P+R)Balances both

Choose the Metric by the Cost of Errors

The deciding question: which error is more expensive?

ProblemMore expensive errorPriority
Cancer screeningMissing a case (FN)Recall
Spam filteringWrongly blocking an important email (FP)Precision
Fraud detectionBoth are costlyF1

Why F1 uses the harmonic mean rather than the arithmetic mean: with P = 1.0 and R = 0.0, the arithmetic mean gives 0.5 (sounds acceptable), while F1 gives 0 — accurately reflecting that the model is useless. F1 does not let you hide a poor metric.

Metrics for Regression Problems

MetricCharacteristics
MAEMean of the absolute errors. Easy to understand, same unit as the output
MSE / RMSESquares the errors — penalizes large errors heavily
R²Compared with the baseline "always predict the mean". 1 is perfect, 0 is no better than the baseline

Choose MAE when all errors are equally bad; choose RMSE when one large error is much more dangerous than many small errors.

from sklearn.metrics import classification_report, confusion_matrix
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))

The Most Common Mistake

Reporting accuracy alone. On imbalanced data, a model that always predicts the majority class achieves very high accuracy without detecting a single case of the class you are looking for.

Study notes

By the end of this lesson you can:

  • Use the confusion matrix as the source of all classification metrics
  • Compute accuracy, precision, recall and F1 and state the question each answers
  • Choose a classification metric by asking which error is more expensive
  • Compare MAE, MSE/RMSE and R-squared and choose between MAE and RMSE
  • Print a full evaluation with classification_report and confusion_matrix

Key idea: Every classification metric comes from the four cells of the confusion matrix; choose the metric according to which error is more costly, and never report accuracy alone.

The confusion matrix: the root of all classification metrics

Predicted +Predicted -
Actual +TPFN (Type II error)
Actual -FP (Type I error)TN

TP and TN are correct predictions; FP is a false alarm and FN is a miss.

Four classification metrics

MetricFormulaQuestion it answers
Accuracy(TP + TN) / totalWhat percentage did we get right?
PrecisionTP / (TP + FP)When the model says positive, how often is it right?
RecallTP / (TP + FN)Of the truly positive cases, how many did the model find?
F12PR / (P + R)Balances both

Example: with TP = 36, FP = 4, FN = 24, TN = 136, precision = 36/40 = 0.90 and accuracy = 172/200 = 0.86.

Choose the metric by the cost of errors

The deciding question is: which error is more expensive?

ProblemMore expensive errorPriority
Cancer screeningmissing a case (FN)recall
Spam filteringwrongly blocking an important email (FP)precision
Fraud detectionboth are costlyF1

Why F1 uses the harmonic mean

With precision = 1.0 and recall = 0.0, the arithmetic mean is 0.5, which sounds acceptable. F1 is 2 * 1.0 * 0 / (1.0 + 0) = 0, which correctly shows that the model is useless: it is never wrong when it says positive, but it finds no positives at all. F1 does not let one good metric hide a poor one.

Metrics for regression problems

MetricCharacteristics
MAEmean of the absolute errors; easy to understand, same unit as the output
MSE / RMSEsquares the errors, so large errors are penalised heavily; RMSE returns to the output's unit
R-squaredcompared with the baseline "always predict the mean"; 1 is perfect, 0 is no better than the baseline

Choose MAE when all errors are equally bad. Choose RMSE when one large error is much more dangerous than many small ones, for example when a single badly wrong demand forecast could empty a warehouse.

One call for a full report

from sklearn.metrics import classification_report, confusion_matrix
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))

classification_report prints precision, recall, F1 and support for every class, plus overall accuracy and averages; confusion_matrix shows the raw counts (rows = actual, columns = predicted).

Worked example

A screening model gives TP = 36, FP = 4, FN = 24, TN = 136 (positive = has the condition).

tp, fp, fn, tn = 36, 4, 24, 136

accuracy = (tp + tn) / (tp + fp + fn + tn)
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
print(round(accuracy, 2), round(precision, 2), round(recall, 2), round(f1, 2))

Expected output: 0.86 0.9 0.6 0.72.

Reasoning step by step:

  1. Accuracy 0.86 looks fine, but 140 of the 200 cases are negative, so accuracy is dominated by the easy negatives.
  2. Precision 0.90: when the model raises an alarm, it is right 9 times out of 10.
  3. Recall 0.60: it misses 40% of the real cases (24 people).
  4. For a screening problem misses are the expensive error, so recall is the metric to improve, even at the cost of some precision. F1 (0.72) summarises the balance.

Common mistakes

  • Reporting accuracy alone. On imbalanced data a model that always predicts the majority class scores very high while finding none of the cases you are looking for.
  • Averaging precision and recall arithmetically. Use F1, which exposes a very low value.
  • Choosing a metric before thinking about error costs. Ask which error is more expensive first.
  • Using MAE when big errors are disastrous. RMSE penalises them more and is the better guide there.

Summary

  • The confusion matrix (TP, FP, FN, TN) is the basis of all classification metrics.
  • Accuracy, precision, recall and F1 each answer a different question.
  • Choose recall when misses are costly, precision when false alarms are costly, F1 when both are.
  • F1's harmonic mean stops one strong metric hiding a weak one.
  • Regression: MAE for equally bad errors, RMSE when large errors are dangerous, R-squared to compare with the mean baseline.
  • classification_report and confusion_matrix give a complete picture in two lines.

Flashcards · Evaluation Metrics

8 words
WordMeaningExample
  • Confusion matrix

    A 2x2 table of TP, FN, FP, TN from which all classification metrics are computed.

    confusion_matrix(y_test, y_pred)

  • Precision

    TP / (TP + FP): how often a positive prediction is correct.

    TP 36, FP 4 -> 0.90

  • Recall

    TP / (TP + FN): the share of truly positive cases the model finds.

    TP 36, FN 24 -> 0.60

  • F1

    2PR / (P + R): the harmonic mean of precision and recall, low if either is low.

    P = 1.0, R = 0 -> F1 = 0

  • Which error is more expensive?

    The deciding question for choosing between precision, recall and F1.

    Cancer screening: FN is worse -> recall.

  • MAE vs RMSE

    MAE treats all errors proportionally; RMSE squares them and penalises large errors heavily.

    Choose RMSE when one big miss is dangerous.

  • R-squared

    Compares the model with always predicting the mean: 1 is perfect, 0 is no better than the mean.

    r2_score(y_test, y_pred)

  • classification_report()

    Prints precision, recall, F1 and support for each class, plus accuracy and averages.

    print(classification_report(y_test, y_pred))