Machine Learning radial basis neural network (RBF NN)

This paper summarizes the notes based on the series of machine learning techniques in Taiwan.

The main content is as follows:

Firstly, the structure of hypothesis and network of radial basis function network is introduced, then the RBF Neural Network learning algorithm is introduced, and the learning by using K-means is studied, and finally the understanding and understanding of this neural network is deepened through an example.

hypothesis and network structure of RBF neural network

We derive the boundary of a large margin from a Gaussian kernel-based support vector machine in an infinite dimension, and this Gaussian kernel is a radio Basis Function Kernel. There are two meanings: one is radio, which represents the distance between X and the center point XM, which can be a certain norm of any distance measurement, such as the L2 norm, and the other is basis, which means the Union (combine) The following is a more rustic definition of the RBF neural network: As shown, the RBF neural network is actually a linear combination of selected radial radial hypothesis.

So why is it a neural network? We look at the structure comparison shown, from which we can see that the hidden layer of the two (hidden layer) is different, that is, the excitation function of the RBF neural network or the mapping is the radial basis function (distance + Gauss), and the output layer is the same, is the characteristic of the transformation of the linear fusion.

This gives the RBF neural network the hypothesis, as shown, if you remember kernel svm hypothesis words (representation theory), The hypothesis of the SVM we get is actually the linear combination of support vectors, that is, the hypothesis that affect our last get are only related to the points of these "representatives". Similarly, the same is true for RBF, where we need to find some central point (center) that can represent the data. So what we're going to do when we're doing training is: Given the radial basis function (RBF) and the output, then determine the coefficient β for these center and linear combinations. Another worthwhile point is to look at the relationship between radial basis function neural networks and other feedforward neural networks from another perspective, as can be seen in my other blog post: Neural network summary (first draft), the key point is from the BP neural network to the radial basis function neural network, One obvious feature is that we do not have to train all the global link weights, but only adjust the weight of some of the important effects of the output, which can significantly improve the training speed of the neural network.

Another important element is the G a The radial basis function of Ussian is actually a measure of similarity, and it is the similarity measurement in the original space. and feature transformation is the transformation of the similarity between distance and center point.

Full RBF Neural networkthe full RBF neural network is actually a very lazy rbf neural network. The reason lazy says it does not do center, that is, it takes all the observed data as the center, that is, the input size is n, then the size of center is also n. A form of uniform that blends the similarity of each input data to all training sample points. As shown in the following:

Nearest neighbor model (Nearest neighbor)

The idea is that we vote for all the output that passes through the hypothesis (vote,aggregation), but instead choose the label of the point closest to the known sample point at the same level as the output. This is the typical nearest neighbor model. So it's easier to look at training, and actually just save the observed data, but the testing process is more complicated, because the sample points of all the known labels are calculated in similarity, then the nearest neighbor is selected and the label of the sample is output. A little expansion, we can still do fusion, we choose the most similar k neighbors, and then by the K-neighbor polling, or linear fusion, and then output, such a model is called the K nearest neighbor model.

Regularization

What happens if we use the full RBF neural network for regression, as shown in the following:

As can be seen, after the above design to train we will get ein=0, which in the interpolation method approximation function is very good, but we know that such a good ein is likely to appear overfitting, so need to carry out regularization design. A common method is to add a constraint or a penalty function. Doing regularization in different spaces has different results. Another way to regularization is to limit the number of center, we use few center instead of full, which can reduce the complexity of the model to some extent. So how do you extract the center point from a bunch of x? What is the standard of extraction?

So the problem turned to the cluster problem cluster problem.

cluster problem cluster problem (K-means)

The mathematical description of the clustering problem is shown by optimizing the cluster error (using squared error).

So it leads to the very famous K-means clustering algorithm, the specific content see my another blog K-means algorithm

application of K-means to RBF neural network

Then the K-means method is used in the RBF neural network, the following process can be obtained

The following are the experimental results using the K-means algorithm process

The full RBF using regularization, the RBF with few center (K-means) and the full RBF experiment based on nearest neighbor are compared, it can be seen that although the full The RBF effect may be better than K-means, but generally it is not often used due to computational complexity and overfitting risk.

*************************************************************************************************************** **********************

For radial basis function neural networks, just grasp the rustic representation of its hypothesis: a bunch of center similarity (Gaussian RBF) Linear fusion (Vote,linear aggregation) is good. The completed feature conversion is the distance to the similarity. The output is a linear combination of similarity.

Then the three more important things that are involved in the middle are: 1. Nearest neighbour thought; 2. K-means algorithm; 3. Alternating optimization's thoughts. It's great to be able to master three of these important things.

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Machine Learning radial basis neural network (RBF NN)