You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
After this unit you will: compute the Gini of a branch and understand why the algorithm chooses the attribute with the smallest Gini.
Gini = 1 − Σ (pᵢ)²
Splitting by age: Gini = (5/14)·0.48 + (4/14)·0 + (5/14)·0.48 = 0.343.
By the end of this lesson you can:
Key idea: Gini impurity measures how mixed a group is (0 = perfectly pure, 0.5 = a 50/50 mix for two classes); the tree picks the feature whose weighted Gini is smallest.
For a group of samples:
Gini = 1 - (p1^2 + p2^2 + ... + pc^2)
Here c is the number of classes and pi is the proportion of samples in class i. In the loan example there are only two classes, approved (Yes) and rejected (No). The recipe is:
The lecture checks the formula on the age groups from the previous lesson.
Middle group (4 Yes, 0 No). p(Yes) = 4/4 = 1, p(No) = 0/4 = 0. Gini = 1 - (1^2 + 0^2) = 1 - 1 = 0. A Gini of exactly 0 means the group is perfectly pure.
Young group (2 Yes, 3 No). p(Yes) = 2/5 = 0.4, p(No) = 3/5 = 0.6. Gini = 1 - (0.16 + 0.36) = 1 - 0.52 = 0.48.
The pattern:
| Mix of Yes/No | Gini |
|---|---|
| all one label (e.g. 4 / 0) | 0 (pure) |
| uneven mix (e.g. 3 / 1) | 1 - (0.75^2 + 0.25^2) = 0.375 |
| close to even (e.g. 2 / 3) | 0.48 |
| perfectly even (e.g. 5 / 5) | 0.5 (maximum for two classes) |
The more evenly mixed a group is, the higher its Gini; the purer it is, the lower the Gini, all the way down to 0.
A feature splits the data into several groups, so we combine their Gini values into one score: a weighted average, where each weight is the proportion of samples that fall into that group. For age (14 applicants):
| Group | Samples | Weight | Gini |
|---|---|---|---|
| young | 5 | 5/14 | 0.48 |
| middle | 4 | 4/14 | 0 |
| senior (3 Yes, 2 No) | 5 | 5/14 | 0.48 |
Gini(age) = (5/14)*0.48 + (4/14)*0 + (5/14)*0.48 = 0.171 + 0 + 0.171 = 0.343.
The weights matter: a large impure group should count more than a tiny impure group.
To find the best root node, the algorithm computes the same weighted Gini for every feature (age, income, student, credit rating) and picks the one with the lowest score, because a lower Gini means purer, more informative groups.
The process then repeats recursively. At each new decision node, the algorithm asks which remaining feature best splits that particular subgroup, and continues until the tree is fully grown or a stopping condition is reached, such as a maximum depth.
Compare two candidate splits of 12 samples (6 Yes, 6 No). Both produce two branches of 6 samples.
Step by step:
The same calculation in Python, using the age split from the lecture:
def gini(counts):
total = sum(counts)
return 1 - sum((c / total) ** 2 for c in counts)
def weighted_gini(branches):
n = sum(sum(b) for b in branches)
return sum(sum(b) / n * gini(b) for b in branches)
age = [[2, 3], [4, 0], [3, 2]] # young, middle, senior: [Yes, No]
print(round(gini([2, 3]), 2)) # 0.48
print(round(weighted_gini(age), 3)) # 0.343
Expected output: 0.48 for the young branch and 0.343 for the age feature, matching the lecture. scikit-learn's DecisionTreeClassifier uses exactly this criterion by default (criterion="gini").
Flashcards · 2.4 Gini impurity
8 wordsA measure of how mixed the labels in a group are: 1 minus the sum of squared class proportions.
The group is perfectly pure: every sample has the same label, so no further split is needed.
The highest Gini possible with two classes is 0.5, reached by an exact 50/50 mix.
The Gini of a feature: the average of its branches' Gini values, each weighted by its share of the samples.
The proportion of samples that fall into a branch, used to weight that branch's Gini.
The tree chooses the feature with the lowest weighted Gini, because it gives the purest groups.
Repeating the best-split search at every new decision node on its own subgroup of data.
A stopping condition that limits how many levels of questions the tree may grow.