Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 2: Core Machine Learning Algorithms

2.4 Gini impurity

Print / Save as PDF

2.4 Gini impurity

After this unit you will: compute the Gini of a branch and understand why the algorithm chooses the attribute with the smallest Gini.

Key points

  • The lower the Gini, the better. Gini = 0 means the branch is completely pure — all samples have the same label, so no further splitting is needed.
  • The algorithm chooses the attribute with the smallest weighted Gini.
  • The Gini of an attribute is the weighted average of the Gini of its branches, where the weights are the proportion of samples in each branch.

Gini = 1 − Σ (pᵢ)²

Splitting by age: Gini = (5/14)·0.48 + (4/14)·0 + (5/14)·0.48 = 0.343.

Study notes

By the end of this lesson you can:

  • Write the Gini impurity formula and explain each part of it
  • Compute the Gini of a single branch from its Yes/No counts
  • Compute the weighted Gini of a whole feature, such as age in the loan dataset (0.343)
  • Explain why the decision tree chooses the feature with the lowest Gini and repeats this at every node

Key idea: Gini impurity measures how mixed a group is (0 = perfectly pure, 0.5 = a 50/50 mix for two classes); the tree picks the feature whose weighted Gini is smallest.

The Gini formula

For a group of samples:

Gini = 1 - (p1^2 + p2^2 + ... + pc^2)

Here c is the number of classes and pi is the proportion of samples in class i. In the loan example there are only two classes, approved (Yes) and rejected (No). The recipe is:

  1. compute the probability of each outcome in the group,
  2. square each probability,
  3. add the squares,
  4. subtract the sum from 1.

Gini of a single branch

The lecture checks the formula on the age groups from the previous lesson.

Middle group (4 Yes, 0 No). p(Yes) = 4/4 = 1, p(No) = 0/4 = 0. Gini = 1 - (1^2 + 0^2) = 1 - 1 = 0. A Gini of exactly 0 means the group is perfectly pure.

Young group (2 Yes, 3 No). p(Yes) = 2/5 = 0.4, p(No) = 3/5 = 0.6. Gini = 1 - (0.16 + 0.36) = 1 - 0.52 = 0.48.

The pattern:

Mix of Yes/NoGini
all one label (e.g. 4 / 0)0 (pure)
uneven mix (e.g. 3 / 1)1 - (0.75^2 + 0.25^2) = 0.375
close to even (e.g. 2 / 3)0.48
perfectly even (e.g. 5 / 5)0.5 (maximum for two classes)

The more evenly mixed a group is, the higher its Gini; the purer it is, the lower the Gini, all the way down to 0.

Weighted Gini of a feature

A feature splits the data into several groups, so we combine their Gini values into one score: a weighted average, where each weight is the proportion of samples that fall into that group. For age (14 applicants):

GroupSamplesWeightGini
young55/140.48
middle44/140
senior (3 Yes, 2 No)55/140.48

Gini(age) = (5/14)*0.48 + (4/14)*0 + (5/14)*0.48 = 0.171 + 0 + 0.171 = 0.343.

The weights matter: a large impure group should count more than a tiny impure group.

Choosing the root and growing the tree

To find the best root node, the algorithm computes the same weighted Gini for every feature (age, income, student, credit rating) and picks the one with the lowest score, because a lower Gini means purer, more informative groups.

The process then repeats recursively. At each new decision node, the algorithm asks which remaining feature best splits that particular subgroup, and continues until the tree is fully grown or a stopping condition is reached, such as a maximum depth.

Worked example

Compare two candidate splits of 12 samples (6 Yes, 6 No). Both produce two branches of 6 samples.

  • Split 1: left 5 Yes / 1 No, right 1 Yes / 5 No.
  • Split 2: left 4 Yes / 2 No, right 2 Yes / 4 No.

Step by step:

  1. Split 1, each branch: 1 - ((5/6)^2 + (1/6)^2) = 1 - 26/36 = 0.278. Weighted: 0.50.278 + 0.50.278 = 0.278.
  2. Split 2, each branch: 1 - ((4/6)^2 + (2/6)^2) = 1 - 20/36 = 0.444. Weighted: 0.444.
  3. Split 1 has the lower Gini, so the tree prefers it.

The same calculation in Python, using the age split from the lecture:

def gini(counts):
    total = sum(counts)
    return 1 - sum((c / total) ** 2 for c in counts)

def weighted_gini(branches):
    n = sum(sum(b) for b in branches)
    return sum(sum(b) / n * gini(b) for b in branches)

age = [[2, 3], [4, 0], [3, 2]]      # young, middle, senior: [Yes, No]
print(round(gini([2, 3]), 2))       # 0.48
print(round(weighted_gini(age), 3)) # 0.343

Expected output: 0.48 for the young branch and 0.343 for the age feature, matching the lecture. scikit-learn's DecisionTreeClassifier uses exactly this criterion by default (criterion="gini").

Common mistakes

  • Choosing the feature with the highest Gini. Gini measures impurity, so lower is better.
  • Taking a simple average of branch Ginis. Each branch must be weighted by its share of the samples (5/14, 4/14, 5/14).
  • Forgetting to square the probabilities. 1 - (0.4 + 0.6) = 0 is wrong; the correct value is 1 - (0.16 + 0.36) = 0.48.
  • Thinking Gini can reach 1 with two classes. For two classes the maximum is 0.5, reached by a perfect 50/50 mix.

Summary

  • Gini = 1 - sum of squared class proportions; c is the number of classes.
  • Gini 0 means a perfectly pure group; 0.5 is the most mixed possible with two classes.
  • A feature's Gini is the weighted average of its branches' Gini values; for age it is 0.343.
  • The tree picks the feature with the lowest weighted Gini as the split.
  • Splitting repeats at every decision node until the tree is grown or a limit such as maximum depth is reached.

Flashcards · 2.4 Gini impurity

8 words
WordMeaningExample
  • Gini impurity

    A measure of how mixed the labels in a group are: 1 minus the sum of squared class proportions.

    2 Yes, 3 No: 1 - (0.4^2 + 0.6^2) = 0.48

  • Gini = 0

    The group is perfectly pure: every sample has the same label, so no further split is needed.

    age = middle: 4 Yes, 0 No.

  • Maximum Gini (two classes)

    The highest Gini possible with two classes is 0.5, reached by an exact 50/50 mix.

    5 Yes, 5 No gives Gini = 0.5.

  • Weighted Gini

    The Gini of a feature: the average of its branches' Gini values, each weighted by its share of the samples.

    (5/14)*0.48 + (4/14)*0 + (5/14)*0.48 = 0.343

  • Branch weight

    The proportion of samples that fall into a branch, used to weight that branch's Gini.

    5 of 14 applicants are young: weight 5/14.

  • Best split rule

    The tree chooses the feature with the lowest weighted Gini, because it gives the purest groups.

    Compare Gini(age), Gini(income), Gini(student), Gini(credit rating).

  • Recursive splitting

    Repeating the best-split search at every new decision node on its own subgroup of data.

    After age, split the young branch by student status.

  • max_depth

    A stopping condition that limits how many levels of questions the tree may grow.

    DecisionTreeClassifier(max_depth=4)