You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
After this unit you will be able to: describe bootstrapping and explain how a Random Forest differs from a single decision tree.
By the end of this lesson you can:
Key idea: A Random Forest trains many decision trees, each on a different bootstrap sample (and with random feature choices), then combines their predictions; the diversity of the trees makes the ensemble more robust and accurate than any single tree.
The lecture compares a single tree with a forest of eight colourful trees. Individual decision trees have a well-known weakness: they are unstable and prone to overfitting. They can learn the training data too specifically, including its noise, and then perform poorly on new, unseen data.
The fix is to build many different trees, each trained slightly differently, and then average or vote across all of them. Combining many individual models to get a better collective result is called an ensemble method, and Random Forest is one of the most popular ensemble methods in machine learning.
An obvious question: if every tree uses the same training data, won't all the trees be identical? The answer is bootstrapping, the key trick behind Random Forest.
Start with the full original dataset, for example the items A, B, C, D, E, F, G. Create several smaller random samples from it, and sample with replacement: after an item is picked, it goes back into the pool, so it can be picked again.
Each sample is a slightly different, randomly duplicated mix of the original data. A sample such as {a, a, d, g} drawn from {a, ..., g} is perfectly valid.
If every tree trained on identical data, they would learn very similar patterns and the point of combining them would be lost: there would be no diversity of opinions. Random Forest creates diversity in two ways:
As a result, each tree ends up a little different. This diversity is exactly what makes the ensemble more robust than a single tree.
This "wisdom of the crowd" idea explains why Random Forest is still one of the most reliable algorithms in practice. In scikit-learn it is RandomForestClassifier for classification and RandomForestRegressor for regression.
A forest of 7 trees is asked about a new loan application. The trees predict: Reject, Accept, Reject, Reject, Accept, Accept, Reject.
The bootstrap step and the vote can be simulated in a few lines:
import numpy as np
from collections import Counter
rng = np.random.default_rng(0)
data = np.array(list("ABCDEFG"))
for i in range(3):
sample = rng.choice(data, size=4, replace=True) # with replacement
print("bootstrap sample", i, sample)
votes = ["Reject", "Accept", "Reject", "Reject", "Accept", "Accept", "Reject"]
print(Counter(votes).most_common(1)[0][0])
Expected output: three random samples of 4 letters in which some letters may repeat and others are missing (the exact letters depend on the seed), followed by Reject, the majority vote.
In scikit-learn the whole pipeline is one object:
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=100, random_state=100)
model.fit(x_train, y_train)
y_pred = model.predict(x_test)
RandomForestClassifier and RandomForestRegressor.Flashcards · 2.5 Bootstrapping and Random Forest
8 wordsA strategy that combines many individual models to produce a better collective prediction than any one of them.
Creating new samples by drawing randomly with replacement from the original dataset.
Each picked item is returned to the pool, so it can be picked again in the same sample.
An ensemble of decision trees, each trained on a different bootstrap sample with random feature choices.
How a classification forest combines trees: the class predicted by the most trees wins.
How a regression forest combines trees: it outputs the mean of all the trees' predictions.
Learning the training data too specifically, including noise, so performance on new data is poor.
The scikit-learn class for Random Forest classification; RandomForestRegressor is the regression version.