Margin and Hard-SVM

Hard-SVM connects large-margin classification to a constrained optimization problem.

Programming

From Separation to Optimization

Hard-SVM turns the search for a good separating hyperplane into a constrained optimization problem. The goal is not merely to find parameters that separate the classes. Among parameter pairs that satisfy the required classification constraint for every training example, Hard-SVM selects the solution associated with the largest margin by minimizing the norm of w.

The order matters: first identify which parameter pairs are acceptable through the constraint, then minimize ‖w‖₂ among those acceptable pairs.

The Closest Training Point

The closest training point is identified using the minimum value of |〈w,xᵢ〉 + b|. This minimum singles out the training example whose position is closest according to the expression used by the formulation. Because this point is the limiting point nearest the separating hyperplane, it determines the smallest separation that the classifier must maintain when considering the training data.

How does the closest training point determine the separation relevant to the hyperplane?

Comparing Candidate Closest Points

Suppose two training examples produce absolute values of 4 and 1 for |〈w,xᵢ〉 + b|. Which example is identified as the closest training point by the stated criterion?

Evaluate the criterion: The formulation compares the absolute values obtained from |〈w,xᵢ〉 + b| for the training examples.

Take the minimum: Between 4 and 1, the minimum is 1.

Identify the point: The example producing 1 is the closest training point according to this criterion.

The training example with value 1 is the closest point under the minimum-|〈w,xᵢ〉 + b| rule.

Reading the Hard-SVM Problem

The Hard-SVM formulation has two logically different parts. Its objective is to minimize ‖w‖₂. Its constraint requires every training example to satisfy the classification condition. The objective chooses among acceptable parameter pairs; it does not define acceptability by itself.

How do the objective of minimizing the norm and the classification constraints combine in Hard-SVM?

Choosing Between Feasible Candidates

Imagine that two parameter pairs both satisfy the classification constraint for every training example. Candidate A has ‖w‖₂ equal to 3, and Candidate B has ‖w‖₂ equal to 5. Which candidate does the Hard-SVM objective prefer?

Check acceptability: Both candidates are assumed to satisfy the constraint for every training example, so both are eligible.

Compare the objective: The objective minimizes ‖w‖₂, so compare 3 with 5.

Select the smaller norm: Candidate A has the smaller norm and is therefore preferred by the objective.

Candidate A is selected among these two feasible candidates.

Why Every Example Matters

The classification constraint is applied to every training example rather than to only one representative point. Each example contributes a requirement that the parameter pair must satisfy. A parameter pair is acceptable only when all of those individual requirements hold simultaneously. The optimization then minimizes ‖w‖₂ within that jointly acceptable set.

How does each training example contribute its own constraint, and what remains after all constraints are considered?

What set of separating-hyperplane parameters satisfies all training-example constraints simultaneously?

Checking the constraint for only one training example
The Hard-SVM constraint must hold for every training example.
Fix: Treat each training example as contributing a requirement, then retain only parameter pairs that satisfy all requirements simultaneously.
Treating the objective as the entire optimization problem
The objective chooses among acceptable pairs; it does not establish acceptability on its own.
Fix: Apply the constraint first and minimize ‖w‖₂ only within the resulting feasible set.

From Solution to Normalized Parameters

Solving the constrained optimization problem produces an optimization solution. Hard-SVM then transforms that solution into the stated normalized output parameters. The important distinction is between the parameters found while solving the optimization and the normalized parameters reported as the final separating-hyperplane description.

How does the optimization solution become the normalized parameter output?

MEDIUM

In your own words, explain the difference between a feasible parameter pair, the optimization solution, and the normalized output parameters.

Hints

A feasible pair satisfies the classification constraint for every training example.
The optimization solution is chosen by minimizing ‖w‖₂ among feasible pairs.
The normalized output parameters are produced by transforming the optimization solution.

Your attempt (not saved)

Summary

The closest training point is identified by the minimum of |〈w,xᵢ〉 + b|.
Hard-SVM minimizes ‖w‖₂, but only among parameter pairs satisfying the classification constraint.
The constraint is applied to every training example, so acceptable pairs must satisfy all example-specific requirements simultaneously.
The optimization solution is transformed into the stated normalized output parameters.
The central reading strategy is constraint first, objective second.

Key Takeaways

The closest training point is selected using the minimum of |〈w,xᵢ〉 + b|.
Hard-SVM finds a separating hyperplane through constrained optimization.
The constraint defines acceptable parameter pairs for every training example.
The objective minimizes ‖w‖₂ only within that feasible set.
The selected optimization solution is transformed into normalized output parameters.