1. Overview of nuclear functions:
The kernel function is popularly used to map the low-dimensional space of vectors to a high-dimensional space by a function, which transforms the nonlinear problem of the low-dimensional space into the linear problem of the high-dimensional space, and then uses the previously mentioned series of linear support vector machines, the common kernel functions are as follows:
Polynomial kernel functions:
Gaussian kernel function:
For example, the target function of a hard interval is:
The target function after the kernel function substitution is:
From this two objective function to find common ground, actually will find that the kernel function is acting on the eigenvector, the beginning of the objective function is to use two eigenvectors, and the next kernel function is to do a function of two eigenvectors, if the kernel function is used in a single operation using three eigenvectors of the calculation, At this point the kernel function will be the kernel function of these three eigenvectors, in fact, is a substitution problem, the two eigenvector dot product operation into a kernel function value, that's all.
2. SMO algorithm Overview:
So the next process is to solve the Lagrange multiplier, W, b, so we introduce SMO (Sequence minimization optimization algorithm)
First, the dual problem to be solved by the SMO algorithm is as follows:
SMO algorithm is a heuristic algorithm, the basic idea: if all the solutions of the variables satisfy the KKT condition of this optimization problem, then the solution of the optimization problem is obtained, the SMO algorithm is actually solving the LaGrand day operator. SMO algorithm sub-problem: First select two variables, and then fixed other variables, for the two variables to build two planning problems, sub-problems from two variables, one is the most serious violation of the KKT condition of the variable, the other is automatically determined by the constraints, and then solve this sub-problem, when the two variables meet the KKT condition, The sub-problem solver is complete. The SMO algorithm is to decompose the problem continuously into this sub-problem until all the solutions of the variables satisfy the KKT condition of the problem, thus ending the algorithm
The sub-problem can be written by the above duality (assuming that two variables alpha1, Alpha2 are selected here):
Note: The objective function of this sub-problem is derived from the dual problem above, in the derivation process will find that there are some items are not, because this time only alpha1 and ALPHA2 are variables, others are considered fixed values, similar to the Min (f (x)) and Min (f (x) +c) The solution x is the same (c is constant)
3. Alpha Calculation formula:
The next few formulas are used for each calculation:
Note: EI is the difference between the value of the sum of G (x) for the predicted value of Xi
This computes a value that is not processed by ALPHA2 because each alpha is represented by a range of, specifically, the following piecewise functions:
The new values for ALPHA1 can be obtained by ALPHA2 as follows:
4. Variable selection:
(1) First, the choice of the first variable, in the SMO algorithm overview I also introduced, is the least satisfied with the kkt condition of this problem, kkt conditions are as follows (Kkt is relative to each sample point is (Xi,yi)):
G (xi) if the above-mentioned formula
The selection of the first variable is the outer loop of the SMO, and in the process of inspection, the first step is to traverse all the sample points that satisfy the 0<alpha<c condition, that is, the support vector points on the interval boundary, to verify whether they meet the kkt condition, and if all are satisfied, whether the entire training set satisfies the KKT condition, Select the least satisfying point as the first variable.
(2) followed by the choice of the second variable: the choice of the second variable is the inner loop of the SMO, first assuming that the first variable is already selected, the second variable is the choice to make the ALPHA2 large enough to change, a simple practice is its corresponding | e1-e2| Max. In the special case, through the above methods can not make the target function has enough to reduce, then use the downward heuristic rule to continue to select ALPHA2, traverse all the interval boundary of the support vector points, and then as a ALPHA2 trial, until the target function has enough to fall, if not found to facilitate the entire training set If it is still missing, discard the alpha1 and continue to select another alpha1 through the outer loop.
(3) Calculate B and Difference ei
Because after each completion of the optimization of two variables, it is necessary to recalculate B and Ei, mainly the EI calculation needs B, the formula is as follows:
[Machine learning & Data Mining] SVM---kernel function