Support Vector Machines for Large Margin Classification

Why Separation Alone Is Not Enough

A hyperplane can separate a training set, but that fact alone does not tell us how much room exists between the training points and the separating boundary. That room matters. If the instances change slightly, a boundary with very little room may no longer separate them, while a boundary with more room can remain effective. Large-margin classification focuses on measuring this room.

Finding the Smallest Distance

The margin of a hyperplane with respect to a training set is the smallest distance from any training point to that hyperplane. To determine it, inspect every training point, consider its distance to the boundary, and select the smallest distance. The closest training point determines the margin; distances from the other points cannot make the margin larger.

Support Vectors at the Boundary of Safety

The training points that are closest to the hyperplane are the points that determine the available room. They are often called support vectors in the context of support vector machines because they support the margin: moving the boundary's closest training point farther away would change the minimum distance, while points already farther away do not determine that minimum.

How Margin Creates Robustness

A large margin means that even the closest training point is relatively far from the hyperplane. That distance provides room for the point to move slightly while the same hyperplane continues to separate the training set. By contrast, when the margin is small, a small perturbation can move the closest point across the boundary or otherwise interfere with separation.

Margin and True Error

The margin of a halfspace does not directly report the exact true classification error. Instead, the true error can be bounded using the margin. The important relationship is that increasing the margin is associated with a smaller bound on true error. This relationship does not depend on the Euclidean dimension.

Mistakes in Margin Reasoning

• Using the average distance instead of the smallest distance

  The margin is controlled by the nearest training point, not by a typical or average point.

  Fix: Choose the smallest distance. In this example, the margin is determined by 2.

• Treating separation as proof of robustness

  A slight perturbation may affect separation when the closest point is very near the hyperplane.

  Fix: Inspect the minimum distance and use it to judge how much room the separation has.

• Calling the margin the exact true error

  The margin provides a quantity in terms of which true error can be bounded; it does not directly report the exact true error.

  Fix: State that a larger margin is associated with a smaller bound on true error.

• Assuming the margin relationship depends on Euclidean dimension

  The source concept states that the relationship between increasing margin and a smaller true-error bound does not depend on the Euclidean dimension.

  Fix: Separate the geometric measurement from assumptions about a specific dimension.

Check Your Understanding

1. The margin is the smallest distance from any training point to the separating hyperplane. The nearest training point determines it, so averaging distances gives the wrong result. A larger margin leaves more room for slight perturbations while preserving separation. For a halfspace, the margin is related to true error through a bound: increasing the margin is associated with a smaller bound, not necessarily a directly known exact error.