You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
Knowing three algorithms is less useful than knowing which problem calls for which algorithm.
| Linear regression | Decision tree | Random Forest | SVM | |
|---|---|---|---|---|
| Interpretable | Very good | Very good | Poor | Poor |
| Non-linear relationships | No | Yes | Yes | Yes (kernel) |
| Needs scaling | Recommended | No | No | Required |
| Sensitive to outliers | High | Low | Low | Medium |
| Small data | Good | Prone to overfitting | Good | Good |
| Very large data | Good | Good | Gradually slower | Slow |
1. Must the decision be explainable?
A bank rejecting a loan application, a hospital suggesting a diagnosis, a hiring decision — these settings are often required by law or regulation to give an explanation. In that case a decision tree or linear regression wins, even if its accuracy is a few percentage points lower.
2. Is the relationship in the data linear or not?
Draw a scatter plot first. If you can clearly see a curve, do not start with a linear model.
3. How large is the data?
SVM becomes noticeably slow once the number of samples reaches the hundreds of thousands. With very large data, linear regression or tree-based models are usually more practical.
Always start with a simple baseline — linear or logistic regression. It gives you a number to compare against. If a complex model does not beat the baseline by a meaningful margin, choose the simple one: it is easier to deploy, maintain, and explain.
Starting with the most powerful model. Without a baseline, you cannot tell whether the complex model actually contributes anything or is just wasting resources.
By the end of this lesson you can:
Key idea: Knowing many algorithms matters less than knowing which problem calls for which one; decide by the problem's requirements and always compare against a simple baseline.
| Linear regression | Decision tree | Random Forest | SVM | |
|---|---|---|---|---|
| Interpretable | very good | very good | poor | poor |
| Non-linear relationships | no | yes | yes | yes (kernel) |
| Needs scaling | recommended | no | no | required |
| Sensitive to outliers | high | low | low | medium |
| Small data | good | prone to overfitting | good | good |
| Very large data | good | good | gradually slower | slow |
How to read it:
1. Must the decision be explainable? A bank rejecting a loan, a hospital suggesting a diagnosis, a hiring decision: these settings are often required by law or regulation to give an explanation. In that case a decision tree or linear regression wins, even if its accuracy is a few percentage points lower.
2. Is the relationship in the data linear or not? Draw a scatter plot first. If you can clearly see a curve, do not start with a linear model.
3. How large is the data? SVM becomes noticeably slow once the number of samples reaches the hundreds of thousands. With a few million rows, it is not a good choice; linear models or tree-based models are usually more practical.
Always start with a simple baseline: Linear Regression for a numeric target, Logistic Regression for a categorical one. It gives you a number to compare against.
If a complex model does not beat the baseline by a meaningful margin, choose the simple one. It is easier to deploy, maintain and explain. Without a baseline, you cannot tell whether a complex model contributes anything or is just wasting resources.
Scenario: a bank needs a loan-approval model. Regulations require it to explain every rejection. It has 50,000 past applications with features such as income, existing debt, age and credit history.
Reasoning step by step:
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
baseline = make_pipeline(StandardScaler(), LogisticRegression())
tree = DecisionTreeClassifier(max_depth=5, random_state=100)
for name, model in [("baseline", baseline), ("tree", tree)]:
model.fit(x_train, y_train)
print(name, round(model.score(x_test, y_test), 3))
Flashcards · Choosing an Algorithm: Assumptions and Use Cases
7 wordsA simple first model (linear or logistic regression) that gives a reference score to beat.
A legal or business need to justify each prediction, which favours trees and linear models.
A relationship where the output does not change in a straight-line way with the inputs.
Whether an algorithm needs features on the same scale: required for SVM, not needed for trees.
How strongly extreme values distort a model; high for linear regression, low for trees.
How training time grows with data size; SVM slows noticeably at hundreds of thousands of rows.
If a complex model does not clearly beat the baseline, keep the simpler model.