Machine Learning Support vector Machine (SVM)

Support vector machine algorithm in deep learning does not fire up 2012 years ago, in machine learning algorithm is a dominant position, the idea is in the two classification or multi-classification tasks, the category of the super-plane can be divided into many kinds, then which kind of classification effect is the best. The concept of support vectors is introduced here, and we can always find some sample points of the class at the sub-interface of several categories, with the perpendicular bisector of these sample points as a planar interface that allows the points in the category to be far enough away from the sub-interface, then this plane is the most ideal segmentation super plane.

Note: SVM, as its name implies, uses only a few support samples to separate two or more classes of samples.

SVM is divided into linearly svm, linear SVM and nonlinear SVM, and the kernel function is used in the latter two.

First, the mathematics in SVM:

1. Convex optimization

2. Nuclear function

3. Lagrange Multiplier Method: Refer to this material, relatively easy to understand http://mp.weixin.qq.com/s/tDLNy-u7cNRGkExRqPVUkg

Second, SVM formula derivation

Basic definitions (in two categories for example):

Split plane:, is the feature space conversion function, simply, =x

The target value belongs to { -1,1}, and the sample is classified as 1 or 1 through sign (Y (x))

1. Linear sub-SVM (the sample itself can be divided linearly)

1.1 Objective function: Traverse all x to find a set of w,b in the x nearest to the split face, so that the nearest x to the line is the maximum distance from the

Deformation achieved:

, S.T.

Finishing in getting the final objective function:

1.2 Solving the objective function, introducing Lagrange multiplier method and Kkt condition:

The w,b derivative of the above equation is 0, and in the original formula, the deformation is the constrained optimization problem of solving alpha:

The resulting w,b and hyper-planes are:

2. SVM with linear support (the sample itself is linearly non-divided)

2.1 The relaxation factor is introduced, and the objective function and constraints become:

Given the relaxation factor, equivalent to the introduction of the regular term,c-> infinity, equivalent to no penalty, all samples of the eye points, not tolerance errors, easy to overfitting,c->0, then the transition bandwidth, tolerance error, model of the ability to be good, can prevent overfitting

2.2 To solve the objective function, the Lagrange multiplier method is still used, and the final objective function is:

Solve the optimal alpha and obtain the W and B.

3. Nonlinear SVM, introducing kernel functions

3.1 Ideas: Non-divided samples, through the kernel function map to the new high-dimensional feature space, so that the sample becomes possible, the common kernel functions have polynomial, Gaussian and sigmoid kernel functions, the formula is as follows:

Generally, it is most prudent to use a Gaussian kernel function when you do not know more information, and the Gaussian function (RBF, radial basis function) is a function that maps to an infinite dimension.

The introduction of the nuclear function to map to the high-dimensional space of fat thin, its value is large, then chubby, its value is small, then thin XI ~ ~

Third, the code implementation of SVM

Import NumPy as NP from SKLEARN.SVM import SVR from sklearn.grid_search import GRIDSEARCHCV # 0.17 Grid_search Import M  Atplotlib.pyplot as Plt if __name__ = = "__main__": N = np.random.seed (0) x = np.sort (np.random.uniform (0,  6, N), axis=0) y = 2*np.cos (x) + 0.1*NP.RANDOM.RANDN (N) x = X.reshape ( -1, 1) model = SVR (kernel= ' RBF ') c = Np.logspace ( -2, 2, ten) gamma= Np.logspace ( -2, 2,) SVR = GRIDSEARCHCV (model, param_grid={' C ': C, ' gamma ': gam MA}, cv=5) Svr.fit (x, y) the optimal parameters of print ' C and gamma: \ n ', svr.best_params_ x_test = Np.linspace (X.min (), X.max (), 100
    ). Reshape ( -1, 1) y_predict = svr.predict (x_test) sp = Svr.best_estimator_.support_ plt.figure (facecolor= ' W ')  Plt.scatter (X[SP], y[sp], c= ' B ', marker= ' o ', label= ' Support Vectors ') Plt.plot (x_test, y_predict, ' r--', label= ' RBF Kernel ') plt.plot (x, Y, ' G ') plt.legend (loc= ' best ') plt.title (' SVR with Grid_search C & Gamma ') Plt. Xlabel (' X ') Plt.ylabEl (' Y ') Plt.grid (True) plt.tight_layout () plt.show () 

Running code can search the grid for optimal Chegamma and get the results of SVM visualized as follows:

To summarize:

1.SVM has two parameters need to be adjusted, one is C, one is, do not know how to match the final classification effect is good, the use of Gaussian function (RBF) is the best, you can use Grid_search to find the optimal set of values of super-parameters, rather than the random parameter

2. The ability of a certain model to be limited, will not have a qualitative leap in the classification results, when the adjustment is not good, you should consider changing a model

3. The parameter is not the main, the choice of the feature and the cleaning of the data are more important to the result of the model