Basic SVM knowledge (1)

1. Introduction

In the field of machine learning and pattern recognition, SVM is definitely another major breakthrough in addition to boosting. In general applications, we use mature libraries such as libsvm, however, we may not be clear about the internal mechanism and some basic knowledge many times, so we will clarify some basic knowledge.

 

2. About dimensions

First, let's look at the lower dimension.

We are familiar with the concept of dimension. dimensions are also called dimensions in mathematics, representing the number of parameters. We say that zero dimension is a point, one dimension is a line, two dimensions are a plane, and three dimensions are three-dimensional.

We jumped out of the perspective of mathematics and rose to the perspective of philosophy. We say that dimensions actually represent the perspective of one thing. Each perspective has a corresponding description, so we digitize this description and combine the descriptions of these perspectives to form a multi-dimensional vector. For example, if we look at a cube and want to look at its length, height, width, and weight, we can use a multi-dimensional vector to describe the cube <,>, here we introduce four dimensions, which are called four-dimensional vectors.

In the field of data mining, we often encounter the concept of multi-dimensional. For example, we regard a movie entry as a thing and regard the comments of different users as different dimensions of the movie, this movie may have tens of thousands or even hundreds of thousands of dimensions. For example, in actual business scenarios, users who like this product on many websites also like this product. Here, we convert it to those who like this movie, the simplest method is the simplest cosine-similarity:

 

Here we need to understand a common sense concept called a dimensional disaster. Dimensional disasters are often used as the best excuse not to process high-dimensional data. A dimension disaster usually refers to the fact that in a high-dimensional space, all data is sparse, which leads to a large deviation in distance calculation in terms of similarity measurement.AlgorithmIt will also become very inefficient.

In this case, we need to perform a dimensionality reduction for the data.

 

3. PCA and SVD

PCA, also known as principal component analysis, is a very common dimension reduction algorithm. In short, PCA is a method to map high-dimensional data to lower-dimensional data.

For the mathematical definition, the original data is transformed to a new coordinate system through an Orthogonal Linear transformation, so that the maximum Variance Dimension of the data falls on the first coordinate, it is called the first main component. Next is the second main component, and so on.

 

This is a classic PCA diagram. We will not discuss the implementation algorithm here.

SVD is almost the most commonly used technology in the field of dimensionality reduction. We split a matrix into three smaller matrices for multiplication:

The result after dimensionality reduction is obtained.

Then, we can use the projection of items in different dimensions to continue to use cosine similarity or other similarity measurement methods to calculate the similarity between the two films.

 

4. Maintenance upgrade

Since we have introduced the dimension disaster and many PCA and SVD dimensionality reduction technologies, why do we need to discuss the dimension escalation here?

In many cases, linear classifier cannot achieve satisfactory results, so we usually adopt two solutions:

A. Combine multiple linear classifiers into a non-linear classifier. For example, the boosting method is an embodiment of this idea.

B. First, perform a non-linear transformation on the data and map X to a different space. The typical method is SVM.

Here is an example:

For example, we now have such a set of data, hollow and solid represent two types of data, we now need to classify this set of data. Obviously, this set of data is not linearly segmented. We can use a "superplane" to separate the plane of the Two-dimensional data from the three-dimensional data to a three-dimensional or even higher-dimensional space:

I didn't process this image. In fact, the red line here should be a superplane, that is, an n-dimensional plane. You can imagine it yourself. An example of csdn is easier to understand:

Here, the red and black lines represent two types of data. This is a one-dimensional data, which is non-linear and can be divided. Then we become a structure similar to a parabolic structure, here you can understand that the straight line is bent, and then it becomes linearly severable. This example is equivalent to raising the data from one dimension to two dimensions.

Let's think from another angle. If we give you a bunch of items, how does the human brain classify these items? We still find some characteristics of these items, such as color, shape, size, touch and so on, and then classify the items according to these features. This is actually a process of first upgrading and then dividing.

 

5. kernal Method

In the previous section, we mentioned that in SVM, we need to increase the dimension of the data. This is the overall idea that the kernel method handles non-linear problems.

First, we define a ing.Phi (X)Ing from a low dimension to a high dimension, and then classifying it through a linear classifier in this space. Through repesenter theorem, we can prove that after the conversion, we perform the dot product operation between two points during training and testing the classifier. We can reach the following table:

 

I am so lazy to draw a bunch of formulas, but I still need to write them .........

 

 

 

Here we only provide two explanations:

 

A. Many people think that at the beginning, I had an f (x) = WPhi (X) So we can directly optimize W. Why bother optimizing to the final step? This problem is that if we define a very simple kernel function, we can solve Phi. However, if we define a Gaussian Kernel, we can solve an infinite multi-dimensional problem at this time. No matter in terms of efficiency or representation, we can't address this problem.Phi (X) For specialized computing.

B. in SVM, Each kernel method corresponds to a transformation, which does not mean that we route data to a new space based on the Phi function, and the data is linearly divided, or using this Phi function will achieve better results and performance. So how to use the kernel function can only be verified by experiments, but traditionally, the use of Gaussian Kernel has better results.

 

Next, let's take things into the question of how to construct core functions.ArticleLet's try again.