Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 3: Model Evaluation and Optimization

3.1 Accuracy and Imbalanced Data

Print / Save as PDF

3.1 Accuracy and Imbalanced Data

After this unit you will be able to: compute the accuracy of a classification model and recognize when that number is misleading.

Key points

  • Binary classification: the model assigns each sample to one of two classes (for example dog / cat); we evaluate it by comparing the predicted labels with the true labels.
  • Accuracy = the proportion of correct predictions out of all samples.
  • Imbalanced data has one overwhelmingly dominant class (the majority class) and one very rare class (the minority class) — for example 99% normal transactions, 1% fraud.
  • A model that always guesses “normal” reaches 99% accuracy but catches 0 fraud cases — high accuracy does not mean the model is good.

Study notes

By the end of this lesson you can:

  • Describe a binary classification problem and compare predicted labels with true labels
  • Compute accuracy as the share of correct predictions
  • Define imbalanced data, the majority class and the minority class
  • Explain why a model with 99% accuracy can still be useless
  • Explain why we need metrics beyond accuracy

Key idea: Accuracy only tells you how often the model is right overall; on imbalanced data a model can score very high accuracy by always predicting the majority class while never finding the minority class you actually care about.

How do we know a model is good?

Module 3 opens with a question we have not answered yet: once a model is trained, how do we know whether it is good, and how do we improve it?

The lecture builds everything on one small running example: seven photographs of dogs and cats. Under the photos there is a row of predictions made by the model (a little robot) and a row of facts, the true labels. This is binary classification: there are exactly two possible categories, dog or cat. To evaluate the model, we compare predictions with true labels, position by position. In the lecture's example, four of the seven predictions were correct and three were wrong.

Accuracy

Accuracy is the simplest metric:

accuracy = number of correct predictions / total number of predictions

With 4 correct out of 7, accuracy = 4/7, about 0.57 or 57%.

Intuitively, 90% accuracy sounds good and 20% sounds clearly bad. But how much accuracy is good enough? The lecture shows that this question has no single answer: it depends on what the data looks like.

When 99% accuracy is useless

Consider a medical model with 99% accuracy. Is it good? Often, no.

Imagine a disease that affects only 1% of patients. A "model" that predicts no disease for every patient, without even looking at their data, is correct 99% of the time, because 99% of patients really do not have the disease. It earns a 99% accuracy score, yet it never identifies a single sick patient, which was the whole point of building it.

The same happens in fraud detection: with 99% normal and 1% fraudulent transactions, always predicting "normal" gives 99% accuracy and catches 0 fraud cases.

Imbalanced data

This situation is called class imbalance:

  • the majority class vastly outnumbers the others (normal transactions, healthy patients);
  • the minority class is rare (fraud, disease).

With imbalanced data, accuracy alone can be dangerously misleading. A model can achieve a high score just by favouring the majority class while completely failing at the task we care about: correctly identifying the minority class. In real problems such as disease or fraud detection, the minority class is usually the more important one.

So we need metrics that show how well the model performs on each class, not just overall. This leads to the confusion matrix and to precision, recall and F1 in the next lessons.

Worked example

A bank's test set has 1,000 transactions: 990 normal (0) and 10 fraudulent (1). A lazy model always predicts 0.

import numpy as np
from sklearn.metrics import accuracy_score

y_true = np.array([0] * 990 + [1] * 10)
y_pred = np.zeros(1000, dtype=int)        # always "normal"

print(accuracy_score(y_true, y_pred))
print(np.mean(y_true == y_pred))           # the same thing by hand
caught = np.sum((y_pred == 1) & (y_true == 1))
print(caught, "of", np.sum(y_true == 1), "frauds caught")

Expected output:

0.99
0.99
0 of 10 frauds caught

Reasoning step by step:

  1. The model is correct on all 990 normal transactions and wrong on all 10 frauds, so accuracy = 990/1000 = 0.99.
  2. It catches none of the frauds: from the bank's point of view, it is worthless.
  3. Before trusting any accuracy number, check the class balance with y.value_counts() (pandas) and compare the score with the "always predict the majority class" accuracy. Here that baseline is already 99%.

A useful rule of thumb: if the test set has 980 majority samples and 20 minority samples, a model with 98% accuracy might be doing nothing more than always predicting the majority class.

Common mistakes

  • Reporting accuracy alone on imbalanced data. A high number can hide a model that never finds the minority class.
  • Judging accuracy without a reference. Always compare it with the accuracy of always predicting the majority class.
  • Thinking the majority class is the important one. Often the rare class (fraud, disease) is exactly what the model is for.
  • Assuming one accuracy threshold is "good" for every problem. What counts as good depends on the data and the cost of errors.

Summary

  • Binary classification assigns each sample to one of two classes; we evaluate by comparing predictions with true labels.
  • Accuracy = correct predictions / all predictions.
  • Imbalanced data has a dominant majority class and a rare minority class.
  • Always predicting the majority class can give 99% accuracy while catching 0 minority cases.
  • We need per-class metrics: the confusion matrix, precision, recall and F1.

Flashcards · 3.1 Accuracy and Imbalanced Data

8 words
WordMeaningExample
  • Binary classification

    A problem with exactly two possible categories for each sample.

    Is this photo a dog or a cat?

  • Accuracy

    The proportion of predictions that are correct out of all predictions.

    4 correct out of 7 -> accuracy about 0.57.

  • Imbalanced data

    A dataset in which one class vastly outnumbers the other(s).

    99% normal transactions, 1% fraud.

  • Majority class

    The class with by far the most samples in an imbalanced dataset.

    Healthy patients in disease screening.

  • Minority class

    The rare class in an imbalanced dataset, often the one the model is built to find.

    Fraudulent transactions (1%).

  • Accuracy paradox

    A useless model can have very high accuracy on imbalanced data by always predicting the majority class.

    Always 'no disease' -> 99% accuracy, 0 patients found.

  • accuracy_score()

    The scikit-learn function that computes accuracy from true and predicted labels.

    accuracy_score(y_true, y_pred)

  • Majority-class baseline

    The accuracy obtained by always predicting the majority class; any real model must beat it.

    990 of 1,000 normal -> baseline accuracy 0.99.