SVM (five) SMO algorithm for support vector machine

One -to-One SMO optimization algorithm (sequential minimal optimization)

The SMO algorithm was proposed by Microsoft's John C. Platt in 1998 and is the fastest two-time planning optimization algorithm, especially for linear SVM and data sparse performance. The best information about SMO is that he himself wrote "Sequential Minimal optimization a Fast algorithm for Training support Vector machines".

I read it, and I'll take a look at the summary of this method on the handout.

First, we go back to the question that has been suspended in front of us, the final optimization problem for even function:

To solve the problem is to find the maximum value W on the parameter, and all is known quantity. C is pre-set by us and known quantity.

In accordance with the idea of ascending coordinates, we first fix all the parameters except the above, and then we find the extremum. Wait a minute, this idea is problematic, because if all parameters other than fixed, then it will no longer be a variable (can be introduced by other values), because the problem is defined

Therefore, we need to select two parameters at a time to do optimization, such as and, at this time can be represented by and other parameters. This brings back to W, and W is just about the function that can be solved.

In this way, the main steps of the SMO are as follows:

This means that the first step is to select a pair and the selection method uses heuristic methods (later). The second step, fixing the addition and other parameters, determines the W extremum condition, by the expression.

SMO is efficient because it is efficient for a parameter optimization process after fixing other parameters.

Here's how it's discussed:

Suppose we select the initial value to satisfy the constraint in the problem. Next, we fix, so w is the function of the and. and meet the conditions:

Because both are known fixed values, for the sake of the aspect, you can mark the Narimi value to the right of the equation.

When and XOR, that is, one is 1, and one is-1, they can be represented as a straight line with a slope of 1. The following figure:

The horizontal axis is that the longitudinal axes are, and both in the rectangle box and also in the straight line, so

，

Similarly, when and when the same number,

，

Then we intend to use the expression:

And then back into the W, you get

After expansion, W can be expressed as. Where A,b,c is a fixed value. In this way, the derivation of W can be obtained, but to ensure satisfaction, we use the representation of the derivation, but the final, according to the following conditions to obtain:

So get the new value after we can get it.

Below enter Platt's article, to find the heuristic search method and the formula of B value.

Here the article uses the symbol is a bit different, but the essence is the same, first to familiarize yourself with the expression of the symbol in the article.

The output function that defines the feature to the result in the article is

is consistent with the substance of our previous.

The original optimization problems were: