Introduction to SVM and SVR

1, Support vector Machine (SVM) is a better way to realize the idea of minimizing the structure risk. Its machine learning strategy is the structure risk minimization principle in order to minimize the expected risk and minimize the empirical risk and confidence range.

The basic idea of support vector machine method:

(1) It is a learning machine specifically for finite sample situations, which realizes the minimization of structural risk: seeking a tradeoff between the accuracy of the approximation of the given data and the complexity of the approximation function, in order to obtain the best generalization ability;

(2) Finally, it solves a convex two-time programming problem, theoretically, it will be the global optimal solution, which solves the problem of local extremum which cannot be avoided in the neural network method.

(3) It transforms the practical problem through the nonlinear transformation to the high dimensional feature space, constructs the linear decision function in the high dimensional space to realize the nonlinear decision function in the original space, solves the dimension problem skillfully, and guarantees the good generalization ability, and the algorithm complexity is independent of the sample dimension.

At present, SVM algorithm has been applied in pattern recognition, regression estimation and probability density function estimation, and the algorithm has surpassed the traditional learning algorithm in efficiency and precision.

For empirical risk R, different loss functions can be used to describe, such as e-insensitive function, quadratic function, Huber function, Laplace function, etc.

Kernel functions generally have polynomial nuclei, Gaussian radial basis nuclei, exponential radial basis nuclei, multi-hidden layer-aware nuclei, Fourier series nuclei, spline cores, B-spline nuclei, and so on, although some experiments show that different kernel functions can produce almost the same results in the classification, but in regression, different kernel functions tend to have a great influence on the fitting results.

2. Support Vector regression algorithm

In order to realize linear regression by constructing linear decision function in high dimensional space after ascending dimension, the base of e-insensitive function is mainly e-insensitive function and kernel function algorithm.

If the fitted mathematical model expresses a curve in a multidimensional space, the result of the E-insensitive function is the "e-pipe" that includes the curve and the training point. In all sample points, only the part of the sample that is distributed on the pipe wall determines the location of the pipe. This part of the training sample is called the "Support vector". In order to adapt to the nonlinearity of the training sample set, the traditional fitting method is usually the height of the step after the linear equation. This method is certainly valid, but the increased adjustable parameters may increase the risk of overfitting. The support vector regression algorithm uses kernel function to solve this contradiction. Using kernel function instead of linear term can make the original linear algorithm "nonlinear", that is, it can do nonlinear regression. At the same time, the introduction of nuclear function achieves the purpose of "ascending dimension", while the adjustable parameter is still controllable through fitting.