Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 2: Core Machine Learning Algorithms

2.7 SVM: Maximum Margin and Support Vectors

Print / Save as PDF

2.7 SVM: Maximum Margin and Support Vectors

After this unit you will be able to: explain the maximum margin and the role of support vectors.

Key Points

  • SVM chooses the hyperplane (the pink line on the slide) so that the margin — the gap between the two dashed yellow lines — is as large as possible.
  • Support vectors are the points closest to the boundary (circled in red on the slide), touching the edge of the margin.
  • The support vectors determine the position of the hyperplane and the width of the margin.
  • Removing a point far from the boundary leaves the hyperplane unchanged; removing a support vector changes it.

Study notes

By the end of this lesson you can:

  • Define the margin and explain why SVM is called a maximum margin classifier
  • Explain, with the driving analogy, why a wide margin generalises better
  • Identify support vectors and explain why only they determine the boundary
  • Describe the practical consequences: memory efficiency and good results on small datasets
  • Explain what the hyperparameter C controls

Key idea: Among all lines that separate the classes, SVM picks the one with the widest margin (empty corridor) between them; only the closest points, the support vectors, hold that boundary in place.

The margin

The lecture answers the question from the previous lesson. The picture now shows:

  • one solid line between the green circles and cyan stars: the chosen decision boundary;
  • two yellow dashed lines parallel to it, one on each side;
  • three points circled in red, labelled support vectors.

The dashed lines mark a corridor on either side of the boundary. The width of that corridor is the margin: a safety buffer of empty space between the two classes, with the boundary running down the middle.

The core idea of SVM: out of all the infinitely many separating lines, choose the one that makes the margin as wide as possible. That is why SVM is called a maximum margin classifier.

Why a wide margin is good

Think of driving a car. If you drive right along the edge of the lane, even a tiny wobble takes you across the line. If you drive down the centre, you have room on both sides to absorb small errors.

A boundary with a wide margin is as far as possible from both groups, so a new data point would have to be quite unusual before it lands on the wrong side. A boundary that hugs one group is fragile; a boundary with a generous margin is robust, and robustness tends to give better performance on new, unseen data.

Support vectors

The red-circled points sit exactly on the edge of the margin, touching the yellow dashed lines. They are the points closest to the boundary, the hardest cases in the dataset.

A surprising fact: the support vectors are the only points that determine where the boundary goes.

  • Move or delete a point far from the boundary (as long as it does not enter the margin) and the boundary does not shift at all.
  • Move a support vector and the boundary moves immediately.

They "support", or hold up, the boundary, hence the name.

Practical consequences:

  • once trained, an SVM only needs to store the support vectors, not the whole training set, so it can be memory efficient;
  • SVMs often work well on relatively small datasets, which is one reason they remain popular in medical or scientific research with only a few hundred carefully collected samples.

Messy data and the hyperparameter C

Real data is rarely this tidy: some points sit on the wrong side and no line separates them perfectly. SVM implementations handle this with the hyperparameter C, which controls the trade-off between a wide margin and tolerating misclassified training points.

CBehaviourTypical effect
smallkeep the margin wide, accept a few mistakesusually generalises better
largeclassify every training point correctly, even with a narrow marginrisk of overfitting, as with deep decision trees

Tuning C is a big part of getting good results with SVM.

Worked example

One-dimensional data: class A has x = 0 and x = 1; class B has x = 5 and x = 6. Where does a maximum margin SVM put the boundary?

  1. The closest points from each class are x = 1 (class A) and x = 5 (class B).
  2. To make the gap as wide as possible, the boundary sits exactly halfway: x = (1 + 5) / 2 = 3.
  3. The margin extends from 1 to 5, so each side is 2 units wide.
  4. The support vectors are x = 1 and x = 5. Deleting x = 0 or x = 6 would not move the boundary.

The same result with scikit-learn:

import numpy as np
from sklearn.svm import SVC

X = np.array([[0], [1], [5], [6]])
y = np.array([0, 0, 1, 1])        # 0 = class A, 1 = class B

model = SVC(kernel="linear", C=1000)   # large C: (almost) no errors allowed
model.fit(X, y)
print(model.support_vectors_.ravel())
print(-model.intercept_[0] / model.coef_[0, 0])

Expected output: the support vectors [1. 5.] and a boundary at about 3.0 (the point where the decision function equals zero).

Common mistakes

  • Thinking every training point shapes the boundary. Only the support vectors do; far-away points have no influence.
  • Confusing the margin with the boundary. The boundary is the solid line; the margin is the width of the empty corridor around it.
  • Always choosing a large C for "higher accuracy". A large C narrows the margin to fit every training point and can overfit.
  • Assuming SVM needs huge datasets. It often works well on small, carefully collected datasets.

Summary

  • The margin is the empty corridor between the two classes, with the boundary in the middle.
  • SVM chooses the separating hyperplane with the widest margin: a maximum margin classifier.
  • A wide margin, like driving in the centre of the lane, makes the model robust to new data.
  • Support vectors are the points on the edge of the margin; only they determine the boundary.
  • The hyperparameter C trades margin width against training errors: small C = wider margin, large C = risk of overfitting.

Flashcards · 2.7 SVM: Maximum Margin and Support Vectors

7 words
WordMeaningExample
  • Margin

    The width of the empty corridor between the decision boundary and the nearest points of each class.

    The gap between the two yellow dashed lines.

  • Maximum margin classifier

    A classifier that chooses, among all separating boundaries, the one with the widest margin; this describes SVM.

    SVM picks the line in the middle of the gap.

  • Support vectors

    The training points closest to the boundary, lying on the margin edge; they alone fix the boundary's position.

    The three red-circled points on the slide.

  • Robust boundary

    A boundary far from both classes, so new points must be unusual to land on the wrong side.

    Driving down the centre of the lane.

  • Hyperparameter C

    SVM setting that trades margin width against training errors: small C allows mistakes, large C narrows the margin.

    SVC(C=0.1) tolerates errors; SVC(C=1000) does not.

  • Memory efficiency of SVM

    A trained SVM only needs its support vectors to make predictions, not the whole training set.

    model.support_vectors_ holds the stored points.

  • SVC

    The scikit-learn class for SVM classification.

    SVC(kernel='linear', C=1.0).fit(X, y)