--SVM analysis of commonly used machine learning algorithms

1, linear-selectable SVM and hard-interval maximum
2, Linear-divided SVM and soft interval (Soft Margin) to maximize
3. Nonlinear SVM and kernel function
4. SMO algorithm ch1, linearly-selectable SVM and hard-interval maximization 1, problem description

1) Problem Description: given a series of samples (X1,y1), (x2,y2),..., (Xn,yn), where Xi is the m-dimensional vector, Yi belongs to { -1,+1}.
2) The final objective: to find a separating surface w*x+b=0, discriminant function f (x) =sign (w*x+b), so that the data in the sample set can be correctly divided;
3) There is a problem: There are multiple hyper-planes that meet the above separation conditions.
4) Solve the idea: to introduce the constraint of maximizing the interval. The interval refers to the nearest (sample) point twice times the distance from the separation plane. 2. function interval and geometric interval

1) function interval:

The interval of the function can indicate the correctness and the degree of certainty of the classification prediction. But when the interval between W and B is increased (twice times), the hyper-plane does not change, but the function interval becomes twice times. So it's going to be normalized. Order | | w| | = 1;

2) Geometry interval

So the above formula can tell the relationship between the function interval and the geometric interval:
3. Maximum Interval

The basic idea of support vector machine is: The super plane that can correctly divide the sample set and the maximum geometry interval. The optimization problem of the derivation constraint:

The above also mentions that with the change of w,b scaling scale, the function interval is changed exponentially, and now the function interval is 1; the optimization problem is equivalent to
equivalent to:

This is a convex two-time planning problem.
Introduce several concepts:
1) Convex set:
A point set (or region), if a segment that joins any of the two points x1,x2 is all contained within the set, it is called a convex set, otherwise it is not a convex set.

2) convexity condition:
(1). Judging the convexity of a function according to the first derivative (the gradient of the function):
F (x) is defined on the convex set R, and has a continuous first-order derivative function, then f (x) on R is a convex function of the necessary and sufficient condition is the convex set R any different two points 1, inequality

Heng set up.
(2). Judging the convexity of a function according to the second derivative (Hesse matrix)
F (x) is defined on the convex set R and has a continuous second derivative function, then f (x) on R is a convex function of the necessary and sufficient condition: the Hesse matrix is semi-positive on R.

3) Convex plan:
For constrained optimization problems

If f (x), GJ (x) is a convex function, then this problem is convex planning.

Now back to our optimization problem, the convex two-time target optimization problem with linear constraints, with global optimal values: 4, specific ideas (linear SVM learning algorithm-Maximum interval method)

1) Constructs and solves the constrained optimal problem: and obtains the w*,b*

2) The separation of super-planar and categorical decision-making functions is obtained:
5, the introduction of Lagrange multiplier AI, and dual properties optimization problem;

6. Specific ideas (linear SVM learning algorithm)

From the above, it is known that w*,b* only relies on sample points (Xi,yi) of ai>0 in the sample, so that the instance Xi is called a support vector.

The two advantages of converting the original problem to a dual problem:
1) Dual problems are often more easily solved;
2) The duality problem can be more naturally introduced into the kernel function, and then extended to the nonlinear classification problem. 2. Linear SVM and maximum soft interval

adjourned