You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
| Predicted + | Predicted − | |
|---|---|---|
| Actual + | TP | FN (Type II error) |
| Actual − | FP (Type I error) | TN |
| Metric | Formula | Question it answers |
|---|---|---|
| Accuracy | (TP+TN)/total | What percentage did we get right? |
| Precision | TP/(TP+FP) | When the model says positive, how often is it right? |
| Recall | TP/(TP+FN) | Of the truly positive cases, how many did the model find? |
| F1 | 2PR/(P+R) | Balances both |
The deciding question: which error is more expensive?
| Problem | More expensive error | Priority |
|---|---|---|
| Cancer screening | Missing a case (FN) | Recall |
| Spam filtering | Wrongly blocking an important email (FP) | Precision |
| Fraud detection | Both are costly | F1 |
Why F1 uses the harmonic mean rather than the arithmetic mean: with P = 1.0 and R = 0.0, the arithmetic mean gives 0.5 (sounds acceptable), while F1 gives 0 — accurately reflecting that the model is useless. F1 does not let you hide a poor metric.
| Metric | Characteristics |
|---|---|
| MAE | Mean of the absolute errors. Easy to understand, same unit as the output |
| MSE / RMSE | Squares the errors — penalizes large errors heavily |
| R² | Compared with the baseline "always predict the mean". 1 is perfect, 0 is no better than the baseline |
Choose MAE when all errors are equally bad; choose RMSE when one large error is much more dangerous than many small errors.
from sklearn.metrics import classification_report, confusion_matrix
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
Reporting accuracy alone. On imbalanced data, a model that always predicts the majority class achieves very high accuracy without detecting a single case of the class you are looking for.
By the end of this lesson you can:
classification_report and confusion_matrixKey idea: Every classification metric comes from the four cells of the confusion matrix; choose the metric according to which error is more costly, and never report accuracy alone.
| Predicted + | Predicted - | |
|---|---|---|
| Actual + | TP | FN (Type II error) |
| Actual - | FP (Type I error) | TN |
TP and TN are correct predictions; FP is a false alarm and FN is a miss.
| Metric | Formula | Question it answers |
|---|---|---|
| Accuracy | (TP + TN) / total | What percentage did we get right? |
| Precision | TP / (TP + FP) | When the model says positive, how often is it right? |
| Recall | TP / (TP + FN) | Of the truly positive cases, how many did the model find? |
| F1 | 2PR / (P + R) | Balances both |
Example: with TP = 36, FP = 4, FN = 24, TN = 136, precision = 36/40 = 0.90 and accuracy = 172/200 = 0.86.
The deciding question is: which error is more expensive?
| Problem | More expensive error | Priority |
|---|---|---|
| Cancer screening | missing a case (FN) | recall |
| Spam filtering | wrongly blocking an important email (FP) | precision |
| Fraud detection | both are costly | F1 |
With precision = 1.0 and recall = 0.0, the arithmetic mean is 0.5, which sounds acceptable. F1 is 2 * 1.0 * 0 / (1.0 + 0) = 0, which correctly shows that the model is useless: it is never wrong when it says positive, but it finds no positives at all. F1 does not let one good metric hide a poor one.
| Metric | Characteristics |
|---|---|
| MAE | mean of the absolute errors; easy to understand, same unit as the output |
| MSE / RMSE | squares the errors, so large errors are penalised heavily; RMSE returns to the output's unit |
| R-squared | compared with the baseline "always predict the mean"; 1 is perfect, 0 is no better than the baseline |
Choose MAE when all errors are equally bad. Choose RMSE when one large error is much more dangerous than many small ones, for example when a single badly wrong demand forecast could empty a warehouse.
from sklearn.metrics import classification_report, confusion_matrix
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
classification_report prints precision, recall, F1 and support for every class, plus overall accuracy and averages; confusion_matrix shows the raw counts (rows = actual, columns = predicted).
A screening model gives TP = 36, FP = 4, FN = 24, TN = 136 (positive = has the condition).
tp, fp, fn, tn = 36, 4, 24, 136
accuracy = (tp + tn) / (tp + fp + fn + tn)
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
print(round(accuracy, 2), round(precision, 2), round(recall, 2), round(f1, 2))
Expected output: 0.86 0.9 0.6 0.72.
Reasoning step by step:
classification_report and confusion_matrix give a complete picture in two lines.Flashcards · Evaluation Metrics
8 wordsA 2x2 table of TP, FN, FP, TN from which all classification metrics are computed.
TP / (TP + FP): how often a positive prediction is correct.
TP / (TP + FN): the share of truly positive cases the model finds.
2PR / (P + R): the harmonic mean of precision and recall, low if either is low.
The deciding question for choosing between precision, recall and F1.
MAE treats all errors proportionally; RMSE squares them and penalises large errors heavily.
Compares the model with always predicting the mean: 1 is perfect, 0 is no better than the mean.
Prints precision, recall, F1 and support for each class, plus accuracy and averages.