Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 2: Core Machine Learning Algorithms

2.8 SVM: The Kernel Trick

Print / Save as PDF

2.8 SVM: The Kernel Trick

After this unit you will be able to: explain how the kernel trick helps SVM handle data that is not linearly separable.

Key Points

  • Some data has two classes that cannot be separated by a straight line in the original space (one class sits in the middle, surrounded by the other).
  • The kernel trick maps the data into a space with more dimensions (for example from a 2D plane up to a 3D cube).
  • In the new space, a hyperplane (the yellow plane on the slide) can separate the two classes.
  • The kernel trick increases the dimensionality to make separation possible; it does not reduce dimensions or remove points.

Study notes

By the end of this lesson you can:

  • Recognise non-linearly separable data, such as a cluster surrounded by a ring of another class
  • Explain how the kernel trick lifts data into a higher-dimensional space where a flat plane separates it
  • Choose between the linear, polynomial and RBF kernels at a practical level
  • Explain why SVM requires feature scaling before training
  • List the five core algorithms of Module 2 and the structure they share

Key idea: When no straight line can separate the classes, the kernel trick lets SVM find a flat separating hyperplane in a higher-dimensional space, which corresponds to a curved boundary in the original space.

When a straight line is impossible

Everything so far assumed the two classes could be separated by a straight line. The lecture now shows a flat two-dimensional plane where the green circles are bunched together in the middle and the cyan stars surround them.

Try to draw one straight line with all circles on one side and all stars on the other: it cannot be done. No straight line separates a cluster in the centre from a ring around it. Such data is called non-linearly separable.

Lifting the data into a higher dimension

On the right of the slide, the same points are lifted into a three-dimensional box. Now a flat yellow plane slices through the box, with the green circles below the plane and the cyan stars above it. In three dimensions, a simple flat surface separates them perfectly. The arrow from the plane to the cube represents this mapping into a space with more dimensions.

This is the kernel trick, one of the most elegant ideas in classical machine learning. You do not need its mathematics for this course; what matters is the practical effect:

  1. the data cannot be separated by a line in the original space;
  2. the kernel maps it into a space with more dimensions;
  3. a flat hyperplane separates the classes in that space;
  4. back in the original space, that hyperplane looks like a curved boundary.

The kernel trick increases the number of dimensions; it does not reduce dimensions or remove points.

Choosing a kernel

You choose the kernel when you set up the model:

KernelWhat it doesWhen to use it
linearthe ordinary straight-line SVMdata close to linearly separable
poly (polynomial)curved boundaries of a chosen degreepolynomial-shaped relationships
rbf (radial basis function, also called Gaussian)very flexible curved boundariesthe most common default for general problems

Always scale features for SVM

SVM is a distance-based algorithm: the margin is all about distances between points. So SVM is sensitive to feature scaling. If one feature is measured in hundreds of millions and another in single digits, the large-scale feature dominates every distance calculation, and the model effectively ignores the other feature.

Always normalise or standardise features before training an SVM. This is not optional; skipping it can turn a good model into a useless one.

Wrapping up Module 2

Module 2 covered five algorithms: Linear Regression, Logistic Regression, decision trees, Random Forest and SVM. Different as they look, they share the same structure: take in features, learn from labelled training examples, and make predictions on new data. Module 3 turns to evaluating and optimising these models.

Worked example

One-dimensional data: class A is x = -2, 0, 2; class B is x = -5 and x = 5. Class A sits in the middle with class B on both sides, so no single threshold on the x-axis separates them.

  1. Add a new feature x^2. The points become (x, x^2): A = (-2, 4), (0, 0), (2, 4); B = (-5, 25), (5, 25).
  2. In this 2D space, a horizontal line x^2 = 12 separates them: every A point has x^2 at most 4 and every B point has x^2 = 25.
  3. Back on the original line, x^2 = 12 means x = -3.46 or x = 3.46: a "curved" (two-part) boundary that a single threshold could not give.

In practice the RBF kernel does this kind of lifting automatically, and a pipeline keeps the scaling step attached to the model:

from sklearn.datasets import make_circles
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_circles(n_samples=200, factor=0.3, noise=0.05, random_state=0)

linear_svm = make_pipeline(StandardScaler(), SVC(kernel="linear"))
rbf_svm = make_pipeline(StandardScaler(), SVC(kernel="rbf"))

print(linear_svm.fit(X, y).score(X, y))
print(rbf_svm.fit(X, y).score(X, y))

Expected output: make_circles creates an inner cluster surrounded by a ring, like the slide. The linear SVM scores only around 0.5 to 0.7 (no straight line works), while the RBF SVM scores close to 1.0.

Common mistakes

  • Thinking the kernel trick reduces dimensions. It maps data into more dimensions so a flat separator exists.
  • Using a linear kernel on clearly ring-shaped data. No straight line can separate a centre cluster from a surrounding ring; try rbf.
  • Skipping standardisation. SVM is distance-based; a large-scale feature will dominate and the model can become useless.
  • Believing you must compute the new coordinates yourself. The kernel handles the mapping inside the algorithm; you only choose the kernel.

Summary

  • Non-linearly separable data, such as a cluster inside a ring, cannot be split by a straight line.
  • The kernel trick lifts data into a higher-dimensional space where a flat hyperplane separates the classes.
  • That flat hyperplane corresponds to a curved boundary in the original space.
  • Common kernels: linear, poly, and rbf (the usual default).
  • SVM is distance-based, so always standardise features before training it.
  • All five Module 2 algorithms learn from labelled examples and predict on new data.

Flashcards · 2.8 SVM: The Kernel Trick

8 words
WordMeaningExample
  • Non-linearly separable data

    Data whose classes cannot be separated by any straight line or flat hyperplane in the original space.

    Green circles in the centre, stars in a ring around them.

  • Kernel trick

    Mapping data into a higher-dimensional space where a flat hyperplane can separate classes that were not separable before.

    Lifting 2D points into a 3D box, split by a plane.

  • Linear kernel

    The ordinary straight-line SVM; suitable when the data is close to linearly separable.

    SVC(kernel='linear')

  • Polynomial kernel

    A kernel that produces curved boundaries of a chosen polynomial degree.

    SVC(kernel='poly', degree=3)

  • RBF kernel

    Radial basis function (Gaussian) kernel; very flexible and the most common default choice for SVM.

    SVC(kernel='rbf')

  • Feature mapping

    Creating new features from old ones so the classes become separable, e.g. adding x squared.

    x -> (x, x^2) separates -2, 0, 2 from -5, 5.

  • Feature scaling for SVM

    Standardising features before SVM so no large-scale feature dominates the distance calculations.

    make_pipeline(StandardScaler(), SVC())

  • Distance-based algorithm

    An algorithm whose decisions depend on distances between points, making it sensitive to feature scale.

    SVM, because its margin is a distance.