Machine learning Combat "5" (svm-Support vector machine)

This blog records "Machine Learning Combat" (machinelearninginaction) learning process, including algorithmic introduction and Python implementation. SVM (Support vector machine)

SVM is a classification algorithm, through the analysis of training set data to find the best separation plane, and then use the flat face to the new data classification. This article introduces the general idea of SVM, through some mathematical deduction to simplify the initial problem, and finally converted to a relatively simple two optimization problems. Linear two classifier

Imagine a set of data points on a two-dimensional plane, grouped into two categories:

Using linear wx+b=0 wx+b=0 (w and x vectors) on the plane to classify the data, the purpose of SVM is to find the best line. This line needs to meet two conditions, one is to completely separate the two types of data, that is, the same type of data fall on one side of the line, and two types of data in the closest line to those points (called support vector) from the line must be as large as possible. The intuitive embodiment in the diagram is that the space between the two sides of the line is as large as possible. Geometry spacing

The distance from point to line (extended to higher order is the distance from point to hyperplane) is called the geometric interval (geometrical margin), the formula is as follows, where the molecular y (wx+b) y (wx+b) is called the function interval:
G=y (wx+b) | | w| | g = \frac{y (wx+b)}{| | w| |}
The y in the upper type represents the category of the data point, the point category above the line is 1, and the bottom is 1, which causes the lines with the wrong classification points to get a negative geometric interval, which is filtered out.
Now we can describe the condition of the optimal line by the geometric interval, where g g is the geometric interval of the nearest point to the line in the data set, and the GI g_i represents the geometric interval of a data point to a line, the problem is described as:
Maxg,s.t.,gi≥g \max G, s.t.,g_i \ge g
That is, the smallest geometric interval in the maximized dataset.
Then continue to simplify the problem, the size of the function interval can be changed by multiplying W to change, the line itself does not change, which means that you can take the appropriate values so that the support vector and line function interval of 1, so that the problem becomes:
max1| | w| |,s.t.,yi (wxi+b) ≥1 \max\frac{1}{| | w| |}, S.t.,y_i (wx_i+b) \ge 1
Further analysis, the formula is equivalent to:
min12| | w| | 2,s.t.,yi (wxi+b) ≥1 \min\frac{1}{2}| | w| | ^2, S.t.,y_i (wx_i+b) \ge 1
This is a two-time optimization problem with constraints, which can be solved by Lagrange duality method.

Lagrangian duality

By introducing the Lagrange multiplier, the condition is unified into the equation to optimize and construct Lagrange function, in which α>0 \alpha>0:
L (w,b,α) =12| | w| | 2−∑i=1n