Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 2: Core Machine Learning Algorithms

2.6 SVM: Which Separating Line to Choose

Print / Save as PDF

2.6 SVM: Which Separating Line to Choose

After this unit you will be able to: state the problem that the Support Vector Machine poses on two-class data.

Key Points

  • The data here has two features x₁, x₂ and two classes (green dots, teal stars).
  • There is more than one straight line that separates the two classes perfectly — the slide draws two lines (orange and purple).
  • So “separating correctly” alone is not enough: we need an additional criterion to pick one line.
  • With 2-dimensional data the boundary is a line; with more dimensions the boundary is a hyperplane.

Study notes

By the end of this lesson you can:

  • Describe the two-feature, two-class setting used to introduce the Support Vector Machine
  • Explain why many different lines can separate the same training data perfectly
  • Explain why 100% training accuracy cannot tell you which line is better
  • Name the boundary in two dimensions (a line) and in higher dimensions (a hyperplane)

Key idea: When several boundaries all separate the training data perfectly, training accuracy cannot choose between them; SVM needs an extra, principled rule for picking the boundary that will work best on new data.

The setting: two features, two classes

The Support Vector Machine (SVM) is the final core algorithm of Module 2. It rests on one elegant idea, introduced here as a question.

The lecture shows a simple two-dimensional chart:

  • the horizontal axis is x1 and the vertical axis is x2, our two features (each data point therefore has two features);
  • a cluster of green circles sits in the upper left, and a cluster of cyan stars in the lower right;
  • the two groups are the two classes, for example approved versus rejected loan applications, or spam versus normal email.

In two dimensions the classification boundary is a straight line. With three features it would be a plane, and with more features a hyperplane.

Two lines, both perfect

Two different straight lines are drawn through the chart: an orange line running steeply and a second line (pink or purple on the slide) at a shallower angle. Both lines separate the classes perfectly: every green circle is on one side and every cyan star on the other. Neither line makes a single mistake on the training data.

In fact, for data that can be separated like this there are infinitely many lines that classify the training data perfectly. So "separating correctly" is not enough to identify one line; we need another criterion.

Why training accuracy cannot decide

From a pure training-accuracy point of view, the two lines are identical: both score 100%. But something feels different. The orange line passes very close to some of the data points, while the other line keeps more distance.

Now imagine a new data point that arrives near that blurry region between the groups. The two lines may well disagree about which class it belongs to. Which one should we trust?

The lecture states the lesson plainly: perfect performance on the training data does not tell you which model will perform better on new, unseen data. This is the same concern that motivates splitting data into training and test sets. A boundary that hugs the training points tightly is fragile; a small change in a new point can push it to the wrong side.

SVM answers the question with a specific rule for choosing the best boundary, which is the topic of the next lesson (maximum margin).

Worked example

A bank uses two features, x1 = income score and x2 = debt score, to separate approved (circles) from rejected (stars) applicants. Two analysts propose boundaries:

  • Line P lies in the middle of the empty space between the two groups, equally far from the nearest circle and the nearest star.
  • Line Q also separates every training point correctly, but passes just next to a few stars.

Reasoning step by step:

  1. Training accuracy: both lines are 100% correct, so this number cannot separate them.
  2. New applicant: a new point lands slightly closer to the stars than any training star did. Line Q, which hugs the stars, may put this point on the wrong side; line P still has room to classify it correctly.
  3. Robustness: line P leaves a buffer on both sides, so small variations in new data are less likely to cross it.
  4. Decision: prefer line P. This intuition, keeping the boundary as far as possible from both classes, is exactly what SVM formalises.

A quick check in code that two different lines can both reach 100% training accuracy:

import numpy as np

X = np.array([[1, 4], [2, 5], [1, 5], [4, 1], [5, 2], [5, 1]])
y = np.array([1, 1, 1, 0, 0, 0])            # 1 = circle, 0 = star

line_a = (X[:, 1] - X[:, 0] > 0).astype(int)        # x2 > x1
line_b = (X[:, 1] - 0.5 * X[:, 0] > 1).astype(int)  # x2 > 0.5*x1 + 1
print((line_a == y).mean(), (line_b == y).mean())

Expected output: 1.0 1.0. Both rules are perfect on the training data, even though they are different lines.

Common mistakes

  • Assuming only one line separates the classes. For separable data there are infinitely many perfect separating lines.
  • Picking the model with the best training accuracy and stopping there. Equal training scores say nothing about performance on unseen data.
  • Calling the boundary a line in every case. It is a line only in two dimensions; in general it is a hyperplane.
  • Forgetting what the axes are. x1 and x2 are features, not classes; the classes are shown by the marker shapes.

Summary

  • SVM is introduced with two features (x1, x2) and two classes (circles and stars).
  • More than one, in fact infinitely many, lines can separate separable training data perfectly.
  • All of these lines have the same 100% training accuracy, so accuracy cannot choose between them.
  • Lines that pass close to data points are fragile for new, unseen points.
  • SVM adds a principled rule (maximum margin) to pick the best boundary; in higher dimensions the boundary is a hyperplane.

Flashcards · 2.6 SVM: Which Separating Line to Choose

7 words
WordMeaningExample
  • Support Vector Machine (SVM)

    A classification algorithm that chooses a separating boundary using a principled rule about distance to the classes.

    Separating approved from rejected loan applications.

  • Decision boundary

    The line or surface that divides the feature space into regions predicted as different classes.

    A straight line in the x1-x2 plot.

  • Feature space (x1, x2)

    The space whose axes are the features; each data point is placed by its feature values.

    A point with two features sits on a 2D chart.

  • Linearly separable data

    Data whose classes can be split perfectly by a straight line or hyperplane.

    Circles upper left, stars lower right.

  • Hyperplane

    The general name for a flat boundary: a line in 2D, a plane in 3D, and its higher-dimensional version.

    With 10 features the SVM boundary is a hyperplane.

  • Training accuracy

    The share of training samples a model classifies correctly; it cannot rank lines that all score 100%.

    Orange and pink lines both reach 100%.

  • Generalisation

    How well a model performs on new, unseen data rather than the data it was trained on.

    A line hugging the stars may misclassify a new point.