Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 2: Core Machine Learning Algorithms

Choosing an Algorithm: Assumptions and Use Cases

Print / Save as PDF

Choosing an Algorithm: Assumptions and Use Cases

Knowing three algorithms is less useful than knowing which problem calls for which algorithm.

Comparison table

Linear regressionDecision treeRandom ForestSVM
InterpretableVery goodVery goodPoorPoor
Non-linear relationshipsNoYesYesYes (kernel)
Needs scalingRecommendedNoNoRequired
Sensitive to outliersHighLowLowMedium
Small dataGoodProne to overfittingGoodGood
Very large dataGoodGoodGradually slowerSlow

Three questions before choosing

1. Must the decision be explainable?

A bank rejecting a loan application, a hospital suggesting a diagnosis, a hiring decision — these settings are often required by law or regulation to give an explanation. In that case a decision tree or linear regression wins, even if its accuracy is a few percentage points lower.

2. Is the relationship in the data linear or not?

Draw a scatter plot first. If you can clearly see a curve, do not start with a linear model.

3. How large is the data?

SVM becomes noticeably slow once the number of samples reaches the hundreds of thousands. With very large data, linear regression or tree-based models are usually more practical.

Working principle

Always start with a simple baseline — linear or logistic regression. It gives you a number to compare against. If a complex model does not beat the baseline by a meaningful margin, choose the simple one: it is easier to deploy, maintain, and explain.

The most common mistake

Starting with the most powerful model. Without a baseline, you cannot tell whether the complex model actually contributes anything or is just wasting resources.

Study notes

By the end of this lesson you can:

  • Compare Linear Regression, decision trees, Random Forest and SVM on six practical criteria
  • Apply the three questions (explainability, linearity, data size) before choosing a model
  • Justify preferring an interpretable model when regulations require explanations
  • Explain why every project should start with a simple baseline

Key idea: Knowing many algorithms matters less than knowing which problem calls for which one; decide by the problem's requirements and always compare against a simple baseline.

Comparison table

Linear regressionDecision treeRandom ForestSVM
Interpretablevery goodvery goodpoorpoor
Non-linear relationshipsnoyesyesyes (kernel)
Needs scalingrecommendednonorequired
Sensitive to outliershighlowlowmedium
Small datagoodprone to overfittinggoodgood
Very large datagoodgoodgradually slowerslow

How to read it:

  • Interpretability: linear models give readable coefficients; a tree gives a readable path. A forest of hundreds of trees and an SVM with a kernel cannot be explained so simply.
  • Scaling: trees split on one feature at a time using thresholds, so the scale does not matter. SVM is distance-based, so scaling is required.
  • Outliers: a linear model minimising squared error is pulled strongly by extreme points; trees isolate them in a branch.
  • Data size: SVM training becomes slow as the number of samples grows into the hundreds of thousands.

Three questions before choosing

1. Must the decision be explainable? A bank rejecting a loan, a hospital suggesting a diagnosis, a hiring decision: these settings are often required by law or regulation to give an explanation. In that case a decision tree or linear regression wins, even if its accuracy is a few percentage points lower.

2. Is the relationship in the data linear or not? Draw a scatter plot first. If you can clearly see a curve, do not start with a linear model.

3. How large is the data? SVM becomes noticeably slow once the number of samples reaches the hundreds of thousands. With a few million rows, it is not a good choice; linear models or tree-based models are usually more practical.

Working principle: start with a baseline

Always start with a simple baseline: Linear Regression for a numeric target, Logistic Regression for a categorical one. It gives you a number to compare against.

If a complex model does not beat the baseline by a meaningful margin, choose the simple one. It is easier to deploy, maintain and explain. Without a baseline, you cannot tell whether a complex model contributes anything or is just wasting resources.

Worked example

Scenario: a bank needs a loan-approval model. Regulations require it to explain every rejection. It has 50,000 past applications with features such as income, existing debt, age and credit history.

Reasoning step by step:

  1. Target: approve or reject, so classification.
  2. Explainability: required by regulation. This points strongly to a decision tree (a readable path such as "debt ratio above X, then credit history short, then reject") or Logistic Regression (readable weights).
  3. Linearity: a scatter plot of income against default rate shows a curve with thresholds, which suits a tree.
  4. Size: 50,000 rows is fine for a tree; an SVM would be slower and hard to explain anyway.
  5. Baseline: fit Logistic Regression first. Then fit a depth-limited tree.
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

baseline = make_pipeline(StandardScaler(), LogisticRegression())
tree = DecisionTreeClassifier(max_depth=5, random_state=100)

for name, model in [("baseline", baseline), ("tree", tree)]:
    model.fit(x_train, y_train)
    print(name, round(model.score(x_test, y_test), 3))
  1. Decision: if the tree is a few points better than the baseline, use the tree. If a Random Forest would add only one more point but no explanation, the regulation makes the tree the right answer.

Common mistakes

  • Starting with the most powerful model. Without a baseline you cannot judge whether its complexity pays off.
  • Ignoring legal and business requirements. A slightly more accurate black-box model may be unusable if decisions must be explained.
  • Using SVM on millions of rows. Training becomes very slow; prefer linear or tree-based models.
  • Choosing a linear model without looking at the data. A quick scatter plot can reveal a curve that a straight line will never fit.

Summary

  • Each algorithm has strengths and weaknesses in interpretability, non-linearity, scaling, outliers and data size.
  • Ask three questions: must it be explained, is it linear, how large is the data?
  • Regulated decisions favour decision trees or linear models.
  • SVM requires scaling and becomes slow on very large datasets.
  • Always begin with a simple baseline and keep the simple model unless a complex one is clearly better.

Flashcards · Choosing an Algorithm: Assumptions and Use Cases

7 words
WordMeaningExample
  • Baseline

    A simple first model (linear or logistic regression) that gives a reference score to beat.

    LogisticRegression score 0.78 vs Random Forest 0.79.

  • Explainability requirement

    A legal or business need to justify each prediction, which favours trees and linear models.

    A bank must explain why a loan was rejected.

  • Non-linear relationship

    A relationship where the output does not change in a straight-line way with the inputs.

    A clear curve in a scatter plot.

  • Scaling requirement

    Whether an algorithm needs features on the same scale: required for SVM, not needed for trees.

    StandardScaler before SVC.

  • Outlier sensitivity

    How strongly extreme values distort a model; high for linear regression, low for trees.

    One mansion price pulls a regression line upward.

  • Scalability

    How training time grows with data size; SVM slows noticeably at hundreds of thousands of rows.

    Prefer trees or linear models for millions of rows.

  • Simplicity principle

    If a complex model does not clearly beat the baseline, keep the simpler model.

    Easier to deploy, maintain and explain.