SVM: The entire SVM algorithm process using kernels (kernel function)

Choose all the samples to be landmarks.

One way is to make all training data landmarks, so that there will be M landmarks (m trainnign data) so that features is an X ( This can be a description of the distance between the trainning data/cross validation data/test data) and the distances between these landmarks.

Landmarks selected to derive a new features vector

Given an x, the features vector is computed by these landmarks, and similar to the previous one, will f0=1;

For x (i) In training data, the kernel function maps to F1 (i), F2 (i) ... FM (i) (M trainning data,m), for Fi (i), calculated by Gaussian kernel function 1

How to minimize cost function to derive parameters--θ

We calculate the new features F (with m+1 dimension) according to the kernel function: M landmarks (M data set), +1 is F0=1

The cost function is somewhat different from the previous, because we have used a new features, so Z is not ΘTX (i) but ΘTF (i), and the subsequent regularization term is m+1 dimension, not θ0 regularization , which is accumulated from θ1-m

It is important to note that when using SVM and kernel functions, instead of using θtθ to represent the sum of the squares of parameters, instead of using the θtmθ(m as a matrix that relies on one of the kernel functions used), for another slightly different distance metric, do not use | | θ| | 2 is minimized, but minimizes the measurement of another type (the variable scale form of the parameter vector θ, which is related to the kernel function), so that the efficient operation of the SVM is done in order to adapt to the large data set, because when our datasets are large, There will be a lot of landmarks, and the dimension of Theta will be very large, so the calculation θ will be very slow, so this optimization.

Can kernels be applied to other algorithms? such as logistic regression? Kernels can be applied to other algorithms, but the computational techniques used for SVM are not well extended to other algorithms (such as logistic regression), so it becomes very slow to apply kernels to the logistic regression.

SVM is compatible with kernels , while logistic regression and kernels are very slow.

Some computational tricks are developed specifically for the use of Kernels's SVM .

How to determine some parameters of SVM (such as the σ2 of C and Gaussian kernel functions)The tradeoff between--bias and variance

C=1/λ,λ is a regularization parameter, when C is large , the lambda is small, then high variance,low bias (overfitting)

when C is small , which means λ is large, it will be high bias, low variance (underfitting)

Σ2 is the parameter of the Gaussian kernel function to indicate whether the change between f (new features) and X is flat or sharp.

large σ2, which indicates that the value of the parameter changes gently with X, so underfitting(High bias,lower variance) appears

A small σ2that indicates that the value of the parameter changes sharply as x changes so that overfitting(high variance,lower bias) appears.

SVM: The entire SVM algorithm process using kernels (kernel function)