You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
After this unit you will be able to: state the problem that the Support Vector Machine poses on two-class data.
By the end of this lesson you can:
Key idea: When several boundaries all separate the training data perfectly, training accuracy cannot choose between them; SVM needs an extra, principled rule for picking the boundary that will work best on new data.
The Support Vector Machine (SVM) is the final core algorithm of Module 2. It rests on one elegant idea, introduced here as a question.
The lecture shows a simple two-dimensional chart:
In two dimensions the classification boundary is a straight line. With three features it would be a plane, and with more features a hyperplane.
Two different straight lines are drawn through the chart: an orange line running steeply and a second line (pink or purple on the slide) at a shallower angle. Both lines separate the classes perfectly: every green circle is on one side and every cyan star on the other. Neither line makes a single mistake on the training data.
In fact, for data that can be separated like this there are infinitely many lines that classify the training data perfectly. So "separating correctly" is not enough to identify one line; we need another criterion.
From a pure training-accuracy point of view, the two lines are identical: both score 100%. But something feels different. The orange line passes very close to some of the data points, while the other line keeps more distance.
Now imagine a new data point that arrives near that blurry region between the groups. The two lines may well disagree about which class it belongs to. Which one should we trust?
The lecture states the lesson plainly: perfect performance on the training data does not tell you which model will perform better on new, unseen data. This is the same concern that motivates splitting data into training and test sets. A boundary that hugs the training points tightly is fragile; a small change in a new point can push it to the wrong side.
SVM answers the question with a specific rule for choosing the best boundary, which is the topic of the next lesson (maximum margin).
A bank uses two features, x1 = income score and x2 = debt score, to separate approved (circles) from rejected (stars) applicants. Two analysts propose boundaries:
Reasoning step by step:
A quick check in code that two different lines can both reach 100% training accuracy:
import numpy as np
X = np.array([[1, 4], [2, 5], [1, 5], [4, 1], [5, 2], [5, 1]])
y = np.array([1, 1, 1, 0, 0, 0]) # 1 = circle, 0 = star
line_a = (X[:, 1] - X[:, 0] > 0).astype(int) # x2 > x1
line_b = (X[:, 1] - 0.5 * X[:, 0] > 1).astype(int) # x2 > 0.5*x1 + 1
print((line_a == y).mean(), (line_b == y).mean())
Expected output: 1.0 1.0. Both rules are perfect on the training data, even though they are different lines.
Flashcards · 2.6 SVM: Which Separating Line to Choose
7 wordsA classification algorithm that chooses a separating boundary using a principled rule about distance to the classes.
The line or surface that divides the feature space into regions predicted as different classes.
The space whose axes are the features; each data point is placed by its feature values.
Data whose classes can be split perfectly by a straight line or hyperplane.
The general name for a flat boundary: a line in 2D, a plane in 3D, and its higher-dimensional version.
The share of training samples a model classifies correctly; it cannot rank lines that all score 100%.
How well a model performs on new, unseen data rather than the data it was trained on.