Implementation of SVM algorithm (i.)

keyword (keywords): SVM support vector machine SMO algorithm for machine learning

assuming that the SVM principle is not very understanding, you can first look at the introduction of the Video , to help understand the very practical, and then go into a little bit to see the few Getting Started articles , the author is very specific, after reading the basis of SVM to understand almost the same, and then buy this "support vector machine Introduction" author is Nello Cristianini and John Shawe-taylor, the electronic industry press. Then the SMO algorithm in the back of the book is basically to understand how the SVM is the same thing, and then write a SVM library, for example, such as the use of tools such as LIBSVM, hehe, almost the same. These are the whole process of learning SVM, and it is also a kind of experience.

The following is a simplified version of the SMO algorithm for SVM, and I will combine the Java code to explain the entire SVM learning training process, known as the train training process. So what is the SMO algorithm?

The purpose of the SMO algorithm is to find a function f (x), which allows us to classify the input data x. Since the classification must have a standard of judgement, for example, there are two cases of a and B, then how can we say that x belongs to Class A or not B? It is necessary to have a border, as if two countries have boundaries, assuming the more obvious the boundary, the easier the distinction, so our goal is to maximize the width of the boundary, so easy to distinguish between a class or Class B.

In SVM, to maximize the boundary, you need to minimize this value:

W: is the reference, the larger the value, the more obvious the boundary
C represents the penalty factor, which assumes that an x belongs to a certain class, but it deviates from the class and runs to the other side of the boundary where the other class goes, and the greater the C indicates the less you want to abandon the point, the narrower the boundary.
Delegates: Loose variables
But the problem seems to be difficult to solve, and because SVM is a convex two-time programming problem, convex two-time programming problem has the optimal solution, then the problem is converted into the following form (Kkt condition):

....... (1)

The AI here is a Lagrangian multiplier (the problem is solved by the Lagrange multiplication number)
For the case of (a), the AI is normally classified, within the boundary (we know the correct classification of the point Yi*f (xi) >=0)
For the case of (b), the AI is the support vector, on the boundary
For the case of (c), it shows that the AI is between two boundaries
and the optimal solution needs to satisfy the KKT condition, that is, satisfying (a) (b) (c) conditions are satisfied
There are a few things that will appear to be unsatisfied:

Yiui<=1 but Ai<c is not satisfied, and originally Ai=c
Yiui>=1 but ai>0 is not satisfied with the original ai=0
Yiui=1 but ai=0 or ai=c is not satisfied, and originally should be 0<ai<c
So to find these AI that do not satisfy kkt, and update these AI, but these AI is also subject to another constraint, that is,

So we have another way to update AI and AJ at the same time, to satisfy the following equation

can be guaranteed and constrained to 0.

using Yi ai+yjaj= constant, eliminating AI, can get a convex two-time programming problem about single-variable AJ, without considering its constraint 0<=aj<=c, can obtain its solution as:

............................................. (2)

Here ....... ....... (3)

represent the old value, and then consider the constraint 0<=aj<=c to get an analytic solution of a:

............ (4)

For

So how do we get AI and AJ?

for the AI, the first multiplier, can be found by the few that meet the Kkt, the second multiplier AJ can find a satisfying condition

.................................................................................... (5)

Update for B:

Under meet conditions: Update B. ............... (6)

Finally, all ai,y and B are updated so that the model is out and then through the function:

............................................................ (7)

The input is x, an array, and each value in the group represents a feature.

Whether the output is Class A or class B. (Positive class or negative class)

Here's the basic code snippet:

/* * Default Input value * c:regularization parameter * tol:numerical tolerance * Max passes */double C = 1; Penalty factor Double tol = 0.01;//tolerance limit value int maxpasses = 5; Represents the maximum number of iterations that did not change the Lagrangian multiplier/* Initialize a[], B, passes */double a[] = new double[x.length];//Lagrange multiplier THIS.A = a;//The multiplier is initialized to 0for (int i = 0; i < x.length; i++) {A[i] = 0;} int passes = 0;while (Passes < maxpasses) {//indicates the number of times the multiplier was changed (basically a pair change) int num_changed_alphas = 0;for (int i = 0; i < X.le Ngth; i++) {//indicates the error of the output and true Yi as determined by A and B at a particular stage (7) Double Ei = Gete (i);/* * The case of an AI that violates the KKT condition is the first one that satisfies the KKT condition: * YI*F (i) >= 1 and Al PHA = = 0 (correct classification) * YI*F (i) = = 1 and 0<alpha < C (support vector on boundary) * YI*F (i) <= 1 and Alpha = = C (between boundaries) * * * * RI = Y[i] * Ei = y[i] * F (i)-y[i]^2 >= 0 * Assume ri < 0 and Alpha < C violates the KKT condition * because the original RI < 0 should correspond to Alpha = C *, RI ; 0 and Alpha > 0 violate the kkt condition * due to the original RI > 0 corresponding should be Alpha =0 */if ((y[i] * Ei <-tol && A[i] < C) | | (y[i] * Ei > Tol && a[i] > 0)) {/* * Ui*yi=1 points on the boundary 0 < A[i] < C * Find max| e1-e2| */int j;/* * Boundalpha means the X-point is on the boundary of the corresponding * Lagrange multiplier a set */if (This.boundAlpha.size () > 0) {//reference formula (5) j = Findmax (Ei, This.boun Dalpha);} else//Assuming no boundaries, just pick one of the J! = i ajj = Randomselect (i);d ouble Ej = Gete (j);//Save current AI and ajdouble Oldai = a[i];d ouble oldaj = a[j]; /* * Calculate the range of the multiplier U, V * Reference formula (4) */double L, h;if (y[i]! = Y[j]) {L = Math.max (0, A[j]-a[i]); H = Math.min (C, c-a[i] + a[j]);} else {L = Math.max (0, A[i] + a[j]-C); H = math.min (0, A[i] + a[j]);} /* * Assuming an ETA equal to 0 or greater than 0 indicates that a optimal value should be on L or u */double eta = 2 * k (i, J)-K (I, I)-K (J, J);//formula (3) if (ETA >= 0) continue;a[j] = a [j]-Y[J] * (EI-EJ)/eta;//formula (2) if (0 < a[j] && A[j] < C) This.boundAlpha.add (j); if (A[j] < L) A[j] = L , else if (A[j] > H) a[j] = H;if (Math.Abs (A[j]-Oldaj) < 1e-5) Continue;a[i] = A[i] + y[i] * y[j] * (Oldaj-a[j]); F (0 < A[i] && A[i] < C) This.boundAlpha.add (i);/* * Calculation b1, B2 * Illumination formula (6) */double B1 = b-ei-y[i] * (A[i]- Oldai) * k (i, I)-y[j] * (A[J)-Oldaj) * k (I, j);d Ouble b2 = b-ej-y[i] * (A[i]-oldai) * k (I, J)-Y[j] * (A[j]-oldaj) * k (J, J); if (0 < a[i] && A[i] < c) B = B1;else if (0 < a[j] && A[j] < C) b = b2;else B = (B1 + b2)/2;num_changed_alphas = Num_changed_ Alphas + 1;}} if (Num_changed_alphas = = 0) {passes++;} else passes = 0;} return new Svmmodel (A, y, b);

After the execution of the results can be, test data is mainly used LIBSVM Heart_scale data.

The correct rate of prediction is above 73%.

It would be better if I changed the kernel function from linear to RBF based.

Finally, when it comes to the SVM algorithm implementation package, there should be very many, including SVM LIGHT,LIBSVM, Matlab itself with the SVM toolkit and so on.

In addition, the complete code, I will upload to csdn to provide download.

Click here to download.

If the understanding is wrong, please correct me! Thank you!

My e-mail:[email protected]

My Other Blogs:

Baidu: Http://hi.baidu.com/futrueboy/home

javaeye:http://futrueboy.javaeye.com/

Csdn:http://blog.csdn.net/techq

Implementation of SVM algorithm (i.)