You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
After this unit you will be able to: explain how the kernel trick helps SVM handle data that is not linearly separable.
By the end of this lesson you can:
Key idea: When no straight line can separate the classes, the kernel trick lets SVM find a flat separating hyperplane in a higher-dimensional space, which corresponds to a curved boundary in the original space.
Everything so far assumed the two classes could be separated by a straight line. The lecture now shows a flat two-dimensional plane where the green circles are bunched together in the middle and the cyan stars surround them.
Try to draw one straight line with all circles on one side and all stars on the other: it cannot be done. No straight line separates a cluster in the centre from a ring around it. Such data is called non-linearly separable.
On the right of the slide, the same points are lifted into a three-dimensional box. Now a flat yellow plane slices through the box, with the green circles below the plane and the cyan stars above it. In three dimensions, a simple flat surface separates them perfectly. The arrow from the plane to the cube represents this mapping into a space with more dimensions.
This is the kernel trick, one of the most elegant ideas in classical machine learning. You do not need its mathematics for this course; what matters is the practical effect:
The kernel trick increases the number of dimensions; it does not reduce dimensions or remove points.
You choose the kernel when you set up the model:
| Kernel | What it does | When to use it |
|---|---|---|
linear | the ordinary straight-line SVM | data close to linearly separable |
poly (polynomial) | curved boundaries of a chosen degree | polynomial-shaped relationships |
rbf (radial basis function, also called Gaussian) | very flexible curved boundaries | the most common default for general problems |
SVM is a distance-based algorithm: the margin is all about distances between points. So SVM is sensitive to feature scaling. If one feature is measured in hundreds of millions and another in single digits, the large-scale feature dominates every distance calculation, and the model effectively ignores the other feature.
Always normalise or standardise features before training an SVM. This is not optional; skipping it can turn a good model into a useless one.
Module 2 covered five algorithms: Linear Regression, Logistic Regression, decision trees, Random Forest and SVM. Different as they look, they share the same structure: take in features, learn from labelled training examples, and make predictions on new data. Module 3 turns to evaluating and optimising these models.
One-dimensional data: class A is x = -2, 0, 2; class B is x = -5 and x = 5. Class A sits in the middle with class B on both sides, so no single threshold on the x-axis separates them.
In practice the RBF kernel does this kind of lifting automatically, and a pipeline keeps the scaling step attached to the model:
from sklearn.datasets import make_circles
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_circles(n_samples=200, factor=0.3, noise=0.05, random_state=0)
linear_svm = make_pipeline(StandardScaler(), SVC(kernel="linear"))
rbf_svm = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
print(linear_svm.fit(X, y).score(X, y))
print(rbf_svm.fit(X, y).score(X, y))
Expected output: make_circles creates an inner cluster surrounded by a ring, like the slide. The linear SVM scores only around 0.5 to 0.7 (no straight line works), while the RBF SVM scores close to 1.0.
rbf.linear, poly, and rbf (the usual default).Flashcards · 2.8 SVM: The Kernel Trick
8 wordsData whose classes cannot be separated by any straight line or flat hyperplane in the original space.
Mapping data into a higher-dimensional space where a flat hyperplane can separate classes that were not separable before.
The ordinary straight-line SVM; suitable when the data is close to linearly separable.
A kernel that produces curved boundaries of a chosen polynomial degree.
Radial basis function (Gaussian) kernel; very flexible and the most common default choice for SVM.
Creating new features from old ones so the classes become separable, e.g. adding x squared.
Standardising features before SVM so no large-scale feature dominates the distance calculations.
An algorithm whose decisions depend on distances between points, making it sensitive to feature scale.