SVM

Support Vector Machine

Linear Support Vector Machine

• Can be visualized
• Accurate when the data is well partitioned

Maximizing Margins

For each data point $(x_1,x_2,y)$, where $y$ is the label of the point (+1/-1)

$$y(w_1{x}_1+w_2{x}_2+b)\ge 1$$ $$\begin{array}{rl}\text{Margin}& =\frac{|(b+1)-(b-1)|}{\sqrt{w_1^2+w_2^2}}\\ & =\frac{2}{\sqrt{w_1^2+w_2^2}}\end{array}$$

To maximize the margins, it is the same as minimizing $w_1^2+w_2^2$

Quadratic Programming

While we have the objective function in quadratic form and constraints in linear form, we can solve for the best result.

But if we have some non-linear objective function or constraints, the result we obtain might not be the best and it might be slow to compute.

That is the reason why we transform the question into minimizing $w_1^2+w_2^2$

Non-linear Support Vector Machine

For n-dimensional space where $n\ge 2$, we can use a hyperplane to divide the space into 2 parts.

If we cannot solve a problem in a $n$ dimensional space, we can transform the data into a higher dimensional space using a “nonlinaer” mapping, then use the Linear Support Vector Machine in the higher dimensional space.