Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 2: Core Machine Learning Algorithms

Support Vector Machines

Print / Save as PDF

Support Vector Machine (SVM)

The idea: not every boundary is equally good

Between two classes of data you can draw infinitely many separating lines. SVM picks the line with the widest margin — that is, the one as far as possible from the nearest points of both classes.

Why is a wide margin good? Because it creates a safety buffer. A new data point that shifts slightly still lands on the correct side; with a boundary hugging the edge, even small noise is enough to cause a misclassification.

Support vectors

Only the points closest to the boundary determine the position of the hyperplane — these are the support vectors. Delete a point far away and the boundary does not change; delete a support vector and the boundary changes immediately.

A practical consequence: SVM depends on only a small fraction of the data, so it is memory-efficient at prediction time.

Kernel trick

When the data cannot be separated by a straight line, SVM maps it into a higher-dimensional space where a separating hyperplane exists. The subtle point: the algorithm never actually computes the coordinates in the new space — it only computes inner products through the kernel function, so the cost stays manageable.

KernelUse when
linearData is linearly separable, many features
rbf (default)Curved boundary, shape unknown in advance
polyPolynomial relationships
from sklearn.svm import SVC
model = SVC(kernel="rbf", C=1.0, gamma="scale")
model.fit(x_train, y_train)

The C parameter controls the tradeoff: large C = heavy penalty for misclassification, narrow margin, prone to overfitting. Small C = tolerate a few errors in exchange for a wide margin and better generalization.

The most common mistake

Forgetting to standardize the data before using SVM. SVM is based on distances, so a feature with a large scale completely dominates all the others. Of all the models covered so far, this is the one most sensitive to feature scale.

Study notes

By the end of this lesson you can:

  • Explain why SVM chooses the widest-margin boundary among all separating lines
  • Identify support vectors and predict what happens when a point is removed
  • Describe the kernel trick and pick a kernel (linear, rbf, poly) for a situation
  • Explain the trade-off controlled by the C parameter
  • Explain why standardising data is essential before SVM

Key idea: SVM picks the separating boundary with the widest safety margin, which depends only on the support vectors; kernels extend it to curved boundaries, and it only works well on standardised features.

Not every boundary is equally good

Between two classes of data you can draw infinitely many separating lines. SVM chooses the line with the widest margin: the one as far as possible from the nearest points of both classes.

A wide margin creates a safety buffer. A new data point that shifts slightly still lands on the correct side. With a boundary hugging one class, even small noise can cause a misclassification. That is why SVM prefers the maximum-margin hyperplane rather than just any line that separates the training data.

Support vectors

Only the points closest to the boundary determine where the hyperplane is: these are the support vectors.

  • Delete a point far from the boundary: the boundary does not change.
  • Delete a support vector: the boundary changes immediately.

Practical consequence: SVM depends on only a small fraction of the data, so it is memory efficient when making predictions.

The kernel trick

When the data cannot be separated by a straight line, SVM maps it into a higher-dimensional space where a separating hyperplane exists. The subtle point: the algorithm never actually computes the coordinates in the new space. It only computes inner products (a measure of similarity between pairs of points) through the kernel function, so the computational cost stays manageable.

KernelUse when
lineardata is linearly separable, many features
rbf (default)curved boundary, shape unknown in advance
polypolynomial relationships
from sklearn.svm import SVC
model = SVC(kernel="rbf", C=1.0, gamma="scale")
model.fit(x_train, y_train)

The C parameter

C controls the trade-off between a wide margin and training errors:

CMarginRisk
largenarrow; heavy penalty for every misclassified training pointoverfitting
smallwide; tolerates a few errorsusually better generalisation

C is a hyperparameter: you set it before training, and it is usually tuned with cross-validation (Module 3).

Standardise first

SVM is based on distances, so a feature with a large scale completely dominates the others. Of all the models covered so far, SVM is the most sensitive to feature scale. Standardising the data (for example with StandardScaler) is the most important preprocessing step before using SVM.

Worked example

A dataset has two features: income (in VND, values in the tens of millions) and num_loans (0 to 5). Compare SVM with and without standardisation.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

raw_svm = SVC(kernel="rbf", C=1.0)
scaled_svm = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0))

raw_svm.fit(x_train, y_train)
scaled_svm.fit(x_train, y_train)
print(raw_svm.score(x_test, y_test), scaled_svm.score(x_test, y_test))

Reasoning step by step:

  1. Without scaling, distances between applicants are almost entirely differences in income; a difference of 3 loans is invisible next to a difference of 10,000,000 VND.
  2. The raw model therefore effectively ignores num_loans and its score usually suffers.
  3. With StandardScaler inside a pipeline, both features have mean 0 and standard deviation 1, so both influence the margin.
  4. Expect the scaled pipeline to score clearly higher whenever num_loans carries useful information. Putting the scaler inside the pipeline also means it is fitted on the training data only.

Common mistakes

  • Forgetting to standardise before SVM. The large-scale feature dominates every distance and the model can become useless.
  • Thinking every training point affects the boundary. Only the support vectors do.
  • Assuming the kernel explicitly builds the higher-dimensional data. It only computes inner products, which keeps the cost manageable.
  • Setting C very large to get perfect training accuracy. A narrow margin fitted to every point tends to overfit.

Summary

  • SVM chooses the maximum-margin hyperplane, giving a safety buffer for new points.
  • Support vectors, the points closest to the boundary, alone determine its position.
  • The kernel trick handles non-linear data via inner products in a higher-dimensional space.
  • linear, rbf (default) and poly kernels fit different boundary shapes.
  • Large C = narrow margin and overfitting risk; small C = wider margin and better generalisation.
  • Always standardise features before SVM.

Flashcards · Support Vector Machines

8 words
WordMeaningExample
  • Widest margin

    SVM's rule for choosing a boundary: the one furthest from the nearest points of both classes.

    A line midway between the two clusters.

  • Support vectors

    The training points closest to the boundary; removing one changes the hyperplane, removing others does not.

    model.support_vectors_

  • Kernel function

    A function computing similarity (inner products) as if data were in a higher-dimensional space, without building that space.

    kernel='rbf'

  • gamma

    An RBF kernel setting that controls how far the influence of a single training point reaches.

    SVC(kernel='rbf', gamma='scale')

  • C parameter

    Penalty for misclassified training points: large C narrows the margin and risks overfitting; small C widens it.

    SVC(C=1.0)

  • Standardisation before SVM

    Rescaling features to mean 0 and standard deviation 1 so no feature dominates SVM's distances.

    make_pipeline(StandardScaler(), SVC())

  • poly kernel

    A kernel for boundaries that follow polynomial relationships between features.

    SVC(kernel='poly', degree=2)

  • Memory efficiency

    SVM needs only its support vectors at prediction time, a small fraction of the training data.

    Store 30 support vectors instead of 5,000 rows.