Process analysis of SMO algorithm for support vector machine

1.SVM final optimization problem for even function

2. Caching of kernel functions

Because the matrix is a symmetric matrix, the footprint in memory can be M (m+1)/2

The mapping relationship is:

[CPP]View PlainCopy

1. #define  OFFSET (x, y)       ((x)  >  (y)  ?  ((((((x) +1) * (x)  >> 1)  +  (y)   :  (((y) +1) * (y)  >> 1)  +  (x))   
2. // ...   
3.     for  (unsigned  i = 0; i < count; ++i)   
4.          for  (unsigned j = 0; j <= &NBSP;I;&NBSP;++J)   
5.              cache[offset (i, j)] = y[i] * y[j] * kernel (X[i],  x[j], dimision);   
6. //...  

3. Solving gradients

Since the alpha value is a variable, the alpha value is derivative and the alpha value is then selected to be optimized by the gradient.

Gradient:

[CPP]View PlainCopy

1. for (unsigned i = 0; i < count; ++i)
2. {
3. Gradient[i] =-1;
4. For (unsigned j = 0; J < count; ++j)
5. Gradient[i] + = Cache[offset (i, J)] * Alpha[j];
6. }

Shut Up w Max, when α is reduced, g the bigger the better. Conversely, g the smaller the better.

4. Constraints of the Sequence minimization method (SMO)

Each time a 2 α value is selected for optimization, the other alpha values are treated as constants, depending on the constraint conditions:

After the optimization:

5. Make selection Rules

Because the range of α is in the interval [0,c], α is bound by α

If the selected and the different number, that is, λ=-1, then the same as the increase and decrease

Assume

If so, you should now select

The above propositions can be translated into (note: and equivalence)

If the selected and the same number, that is, λ=1, then and the increase and decrease of the differences

If, then, it should be selected at this time,

The above propositions can be translated into (note: and equivalence)

Or

The above conclusions can be collated (for the sake of simplicity here only the symbol before G and the symbol of y is different)

[CPP]View PlainCopy

2. unsigned x0 = 0, x1 = 1;
4. Optimized alpha value based on gradient selection
6. {
8. Double gmax =-dbl_max, gmin = Dbl_max;
10. For (unsigned i = 0; i < count; ++i)
12. {
14. if ((Alpha[i] < C && Y[i] = = POS | | alpha[i] > 0 && y[i] = NEG) &&-y[i] * Gradient[i] > Gmax)
16. {
18. Gmax =-y[i] * Gradient[i];
20. x0 = i;
22. }
24. Else if ((Alpha[i] < C && Y[i] = = NEG | | alpha[i] > 0 && y[i] = POS) &&-y[i] * GRA Dient[i] < gmin)
26. {
28. Gmin =-y[i] * Gradient[i];
30. x1 = i;
32. }
34. }
36. }

6. Start the solution

Alpha requires that the alpha value of the non-conforming condition be adjusted within the interval [0,c], with the following adjustment rules.

In 2 cases, if λ=-1, namely:

After substituting:

[CPP]View PlainCopy

2. if (y[x0]! = y[x1])
4. {
6. Double coef = cache[offset (x0, x0)] + cache[offset (x1, x1)] + 2 * cache[offset (x0, X1)];
8. if (coef <= 0) Coef = dbl_min;
10. Double delta = (-gradient[x0]-gradient[x1])/coef;
12. double diff = alpha[x0]-alpha[x1];
14. ALPHA[X0] + = Delta;
16. ALPHA[X1] + = Delta;
18. unsigned max = x0, min = x1;
20. if (diff < 0)
22. {
24. max = x1;
26. min = x0;
28. diff =-diff;
30. }
32. if (Alpha[max] > C)
34. {
36. Alpha[max] = C;
38. Alpha[min] = C-diff;
40. }
42. if (Alpha[min] < 0)
44. {
46. Alpha[min] = 0;
48. Alpha[max] = diff;
50. }
52. }

If λ=1, that is:

[CPP]View PlainCopy

2. Else
4. {
6. Double coef = cache[offset (x0, x0)] + cache[offset (x1, x1)]-2 * cache[offset (x0, X1)];
8. if (coef <= 0) Coef = dbl_min;
10. Double delta = (-gradient[x0] + gradient[x1])/coef;
12. double sum = alpha[x0] + alpha[x1];
14. ALPHA[X0] + = Delta;
16. ALPHA[X1]-= Delta;
18. unsigned max = x0, min = x1;
20. if (alpha[x0] < alpha[x1])
22. {
24. max = x1;
26. min = x0;
28. }
30. if (Alpha[max] > C)
32. {
34. Alpha[max] = C;
36. Alpha[min] = sum-c;
38. }
40. if (Alpha[min] < 0)
42. {
44. Alpha[min] = 0;
46. Alpha[max] = sum;
48. }
50. }

Then adjust the gradient, adjust the formula as follows:

[CPP]View PlainCopy

1. for (unsigned i = 0; i < count; ++i)
2. Gradient[i] + = Cache[offset (i, x0)] * delta0 + cache[offset (i, X1)] * DELTA1;

7. Calculation of weights

The calculation formula is as follows:

[CPP]View PlainCopy

2. Double Maxneg =-dbl_max, Minpos = Dbl_max;
4. SVM *SVM = &bundle->svm;
6. for (unsigned i = 0; i < count; ++i)
8. {
10. Double WX = Kernel (svm->weight, data[i], dimision);
12. if (y[i] = = POS && minpos > WX)
14. Minpos = WX;
16. Else if (y[i] = = NEG && Maxneg < WX)
18. Maxneg = WX;
20. }
22. Svm->bias =-(Minpos + Maxneg)/2;

Code Address: http://git.oschina.net/fanwenjie/SVM-iris/

Process analysis of SMO algorithm for support vector machine