You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
After this unit you will be able to: compute the accuracy of a classification model and recognize when that number is misleading.
By the end of this lesson you can:
Key idea: Accuracy only tells you how often the model is right overall; on imbalanced data a model can score very high accuracy by always predicting the majority class while never finding the minority class you actually care about.
Module 3 opens with a question we have not answered yet: once a model is trained, how do we know whether it is good, and how do we improve it?
The lecture builds everything on one small running example: seven photographs of dogs and cats. Under the photos there is a row of predictions made by the model (a little robot) and a row of facts, the true labels. This is binary classification: there are exactly two possible categories, dog or cat. To evaluate the model, we compare predictions with true labels, position by position. In the lecture's example, four of the seven predictions were correct and three were wrong.
Accuracy is the simplest metric:
accuracy = number of correct predictions / total number of predictions
With 4 correct out of 7, accuracy = 4/7, about 0.57 or 57%.
Intuitively, 90% accuracy sounds good and 20% sounds clearly bad. But how much accuracy is good enough? The lecture shows that this question has no single answer: it depends on what the data looks like.
Consider a medical model with 99% accuracy. Is it good? Often, no.
Imagine a disease that affects only 1% of patients. A "model" that predicts no disease for every patient, without even looking at their data, is correct 99% of the time, because 99% of patients really do not have the disease. It earns a 99% accuracy score, yet it never identifies a single sick patient, which was the whole point of building it.
The same happens in fraud detection: with 99% normal and 1% fraudulent transactions, always predicting "normal" gives 99% accuracy and catches 0 fraud cases.
This situation is called class imbalance:
With imbalanced data, accuracy alone can be dangerously misleading. A model can achieve a high score just by favouring the majority class while completely failing at the task we care about: correctly identifying the minority class. In real problems such as disease or fraud detection, the minority class is usually the more important one.
So we need metrics that show how well the model performs on each class, not just overall. This leads to the confusion matrix and to precision, recall and F1 in the next lessons.
A bank's test set has 1,000 transactions: 990 normal (0) and 10 fraudulent (1). A lazy model always predicts 0.
import numpy as np
from sklearn.metrics import accuracy_score
y_true = np.array([0] * 990 + [1] * 10)
y_pred = np.zeros(1000, dtype=int) # always "normal"
print(accuracy_score(y_true, y_pred))
print(np.mean(y_true == y_pred)) # the same thing by hand
caught = np.sum((y_pred == 1) & (y_true == 1))
print(caught, "of", np.sum(y_true == 1), "frauds caught")
Expected output:
0.99
0.99
0 of 10 frauds caught
Reasoning step by step:
y.value_counts() (pandas) and compare the score with the "always predict the majority class" accuracy. Here that baseline is already 99%.A useful rule of thumb: if the test set has 980 majority samples and 20 minority samples, a model with 98% accuracy might be doing nothing more than always predicting the majority class.
Flashcards · 3.1 Accuracy and Imbalanced Data
8 wordsA problem with exactly two possible categories for each sample.
The proportion of predictions that are correct out of all predictions.
A dataset in which one class vastly outnumbers the other(s).
The class with by far the most samples in an imbalanced dataset.
The rare class in an imbalanced dataset, often the one the model is built to find.
A useless model can have very high accuracy on imbalanced data by always predicting the majority class.
The scikit-learn function that computes accuracy from true and predicted labels.
The accuracy obtained by always predicting the majority class; any real model must beat it.