You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
| Component | Role |
|---|---|
| Root node | The root node — the first question |
| Decision node | A node in the middle that asks about a condition |
| Leaf node | The leaf node — the final conclusion |
The biggest strength: you can read the path that leads to each decision. For a rejected loan application, the tree tells you exactly which condition ruled it out — something a bank is required to be able to explain to the customer.
Gini = 1 − Σ (pᵢ)²
Gini measures how "mixed" a branch is:
The algorithm tries each attribute, computes the weighted Gini of the child branches, then picks the attribute with the smallest Gini.
In the loan-approval example from the lecture: splitting on age gives Gini = (5/14)(0.48) + (4/14)(0) + (5/14)(0.48) = 0.343.
from sklearn.tree import DecisionTreeClassifier, plot_tree
model = DecisionTreeClassifier(max_depth=4, random_state=100)
model.fit(x_train, y_train)
plot_tree(model, feature_names=x.columns, filled=True)
A tree with no depth limit keeps splitting until each leaf holds a single sample — that is, it memorizes. Accuracy on train reaches 100%, but on test it is poor.
Ways to control it: max_depth, min_samples_leaf, min_samples_split.
Random Forest trains many trees on subsets sampled with replacement (bootstrapping), then takes a majority vote. The random mistakes of individual trees cancel each other out, so the forest is far more stable than a single tree — at the price of losing most of the interpretability.
Letting the tree grow freely and then being surprised that train accuracy is 100%. That is not a good model, that is a lookup table.
By the end of this lesson you can:
max_depth, min_samples_leaf and min_samples_splitKey idea: A decision tree is the easiest model to explain, but left to grow freely it memorises the training data; depth limits, or a whole forest of trees, keep it useful on new data.
| Component | Role |
|---|---|
| Root node | the first question |
| Decision node | a node in the middle that asks about a condition |
| Leaf node | the final conclusion |
The biggest strength of a tree is that you can read the path that leads to each decision. For a rejected loan application, the tree shows exactly which condition ruled it out. Banks are often required to explain such decisions to customers, which makes trees attractive in regulated settings.
Gini = 1 - sum of (pi)^2, where pi is the proportion of samples of class i in a branch. It measures how mixed a branch is:
| Branch | Calculation | Gini |
|---|---|---|
| 4 Yes, 0 No | 1 - 1^2 | 0 (pure) |
| 3 Yes, 2 No | 1 - (0.6^2 + 0.4^2) | 0.48 |
| 50/50 | 1 - (0.5^2 + 0.5^2) | 0.5 (most mixed) |
The algorithm tries each attribute, computes the weighted Gini of the child branches (each branch weighted by its share of samples), and picks the attribute with the smallest value. In the loan-approval example from the lecture, splitting on age gives (5/14)(0.48) + (4/14)(0) + (5/14)(0.48) = 0.343.
from sklearn.tree import DecisionTreeClassifier, plot_tree
model = DecisionTreeClassifier(max_depth=4, random_state=100)
model.fit(x_train, y_train)
plot_tree(model, feature_names=x.columns, filled=True)
plot_tree draws the learned tree so you can read every question and leaf.
A tree with no depth limit keeps splitting until every leaf holds a single sample: it memorises the training data. Train accuracy reaches 100%, but test accuracy is poor, because the tree has learned noise rather than general patterns.
Ways to control it:
| Hyperparameter | Effect |
|---|---|
max_depth | limits how many levels of questions the tree may ask |
min_samples_leaf | each leaf must contain at least this many samples |
min_samples_split | a node needs at least this many samples before it may split |
If a tree scores 100% on train but much less on test, max_depth is usually the first parameter to adjust.
Random Forest trains many trees on subsets sampled with replacement (bootstrapping) and takes a majority vote. The random mistakes of individual trees cancel each other out, so the forest is far more stable than one tree. The price: you lose most of the interpretability, because you can no longer read one simple path.
A tree reaches 100% accuracy on the training set and 71% on the test set. Diagnose and fix it.
from sklearn.tree import DecisionTreeClassifier
for depth in [None, 2, 4, 6]:
tree = DecisionTreeClassifier(max_depth=depth, random_state=100)
tree.fit(x_train, y_train)
print(depth, tree.score(x_train, y_train), tree.score(x_test, y_test))
None the train score is 1.0 and the test score is the lowest or close to it; as depth is limited, the train score falls while the test score usually rises, and the gap shrinks.max_depth, min_samples_leaf, min_samples_split.Flashcards · Decision Trees
8 wordsThe sequence of questions from the root to a leaf that explains one prediction.
1 minus the sum of squared class proportions in a branch; 0 is pure, 0.5 is a 50/50 mix.
A scikit-learn function that draws a fitted decision tree with its questions and leaves.
An unlimited tree memorises training rows, scoring 100% on train but poorly on test.
The maximum number of levels a tree may grow; the first parameter to adjust when a tree overfits.
The minimum number of samples every leaf must contain, which prevents tiny, over-specific leaves.
The minimum number of samples a node needs before it is allowed to split.
Random Forest gains stability over a single tree but loses most of its readability.