Radial basis networks (RBF network)

Source: http://blog.csdn.net/zouxy09/article/details/13297881

1. Radial basis function

The radial basis function (Radical Basis FUNCTION,RBF) method was proposed by Powell in 1985. The so-called radial basis function, in fact, is some kind of scalar function along radial symmetry. A monotone function, usually defined as a Euclidean distance between any point in space X to a central C, can be recorded as K (| | x-c| |), the effect is often local, that is, when the x away from C function value is very small. For example, a Gaussian radial basis function:

The birth of radial basis function is mainly to solve the problem of multi-variable interpolation. You can look at the diagram below. In particular, a base function is placed on each sample, and each blue point in the figure is a sample, and then the green dashed line in the middle of the graph corresponds to a Gaussian function for each training sample (the center of the Gaussian function is the sample point). Then assuming that the curve of the actual fitting of these training data is the blue one (the right-most figure), if we have a new data x1, we want to know how much it corresponds to F (x1), which is the ordinate of point A. Then the graph can see that the ordinate of point A is equal to the ordinate of the B point plus the ordinate of the C point. While the ordinate of B is the value of the Gaussian function of the first sample point multiplied by a large point of merit, the ordinate of C is the value of the Gaussian function of the second sample point multiplied by the right of the other small point to be worth it. The weights of the other sample points are all 0, because we want to interpolate the point X1 between the first and second sample points, away from the other sample points, then the most important effect of interpolation is close to the point, away from the far no contribution. So the function values of the X1 point are determined by the nearby B and C points. Expand to any new X, these red Gaussian functions multiplied by a weight and then added to the corresponding X place, you can perfectly fit the real function curve.

Two, radial basis network

By the year 1988, Moody and Darken proposed a neural network structure, RBF neural network, which belongs to the forward neural network type, which can approximate any continuous function with arbitrary precision, which is especially suitable for solving the classification problem.

The structure of RBF network is similar to multilayer Feedforward network, it is a three-layer forward network. The input layer is composed of the signal source node, the second layer is the hidden layer, and the hidden element number is determined by the need of the described problem, and the transform function of the hidden element is the RBF radial basis function, which is a non-negative nonlinear function of radial symmetry and attenuation of the center point; The transformation from the space of the human to the hidden layer is nonlinear, and the spatial transformation from the hidden layer to the output layer is linear.

The basic idea of RBF network is: using RBF as the "base" of hidden elements to form the hidden layer space, so that the input vectors can be mapped directly to the hidden space (i.e., no need to pass through the right connection). According to the cover theorem, the non-divided data in the low dimensional space is more likely to become available in the high dimensional space. In other words, the function of the hidden layer of RBF network is to map the input of low dimension space to a high dimensional space by nonlinear function. The curve is then fitted in this high-dimensional space. It is equivalent to looking for a surface that can best fit the training data in an implicit high-dimensional space. This is different from the normal multilayer perceptron MLP.

When the center point of the RBF is determined, the mapping relationship is determined. The mapping of the hidden layer space to the output space is linear, that is, the output of the network is the linear weighting of the output of the hidden unit, and the right here is the network tunable parameter. Thus, from the overall view, the network from the output of the mapping is non-linear, and the network output on the tunable parameters are linear. Thus, the power of the network can be solved directly by the linear equations, which greatly accelerates the learning speed and avoids the local minimum problem.

It can be understood from another aspect that the implicit node basis function of multilayer perceptron (including BP neural network) is linear, and the activation function uses the sigmoid function or the hard limit function. The base function of the hidden node of the RBF network takes a distance function (such as Euclidean distance) and uses a radial basis function (such as the Gaussian function) as the activation function. Radial basis functions a central point of the n-dimensional space has radial symmetry, and the farther the neuron's input is from the center point, the less the neuron activates. This characteristic of hidden nodes is often referred to as "local characteristics".

Design and solution of RBF Network

The design of RBF mainly consists of two aspects, one is the structure design, that is to say, the hidden layer contains several nodes suitable. The other is the parameter design, which is to solve the network parameters. From the above input to output network mapping function formula can be seen, the network parameters mainly include three kinds: radial basis functions of the center, variance and hidden layer to the output layer weights. So far, there have been many methods to solve these three kinds of parameters, mainly can be divided into the following two major categories:

1. Method One:

The center and variance of radial basis functions are obtained by unsupervised method, and the weights of hidden layers to the output layer are obtained through the supervised method (minimum mean square error). Specific as follows:

(1) Randomly select H samples as the center of the H radial basis function in the training sample set. A better approach is to use clustering, such as K-means clustering, to get an H cluster center, which is the center of H for radial basis functions.

(2) When the base function of the RBF neural network is a Gaussian function, the variance can be solved by the following formula:

The Cmax is the maximum distance between the selected centers, and H is the number of hidden layer nodes. Extended constants are calculated so that the radial basis function is not too sharp or peaceful.

(3) The connection weights of the neurons between the hidden layer and the output layer can be calculated directly with the minimum mean square error LMS, the formula is as follows: (Calculate pseudo-inverse) (d is the output value we expect)

2, Method Two:

The supervised learning algorithm is used to train all parameters of the network (the center of the radial basis function, the variance and the weight of the hidden layer to the output layer). The main focus is on the cost function (mean square error) gradient descent, and then modify each parameter. Specific as follows:

(1) The weights of the center, variance and hidden layer to the output layer of the radial basis function are randomly initialized. Of course, you can also use method one (1) to initialize the center of the radial basis function.

(2) through the gradient descent to the network of three parameters are supervised training optimization. The cost function is the mean square error of the network output and the desired output:

Then each iteration, in the negative direction of the error gradient has a certain learning rate adjustment parameters.

Four, the code implementation:

1. The first method

The first method in Zhangchaoyang's blog has a C + + implementation, but the above is for scalar data (both input and output are one-dimensional). And in MATLAB also provides the first method of the improved version (hehe, personally think, you can run in MATLAB open NEWRB view the source code).

One of the functions provided by MATLAB is NEWRB (). It has the ability to automatically increase the number of hidden neurons in the network until the mean variance satisfies our required precision or the number of neurons reaches the maximum (that is, the number of samples we provide, when the number of neurons is the same as our sample number, the RBF network now has a mean square error of 0). It can also be used in a simple way:

RBF = NEWRB (train_x, train_y);

Output = RBF (test_x);

Just give the training sample to it to get a RBF network. Then we give it the input to get the output of the network.

2. The second method

The second method on Zhangchaoyang's blog also has a C + + implementation, but the above is for scalar data (both input and output are one-dimensional). But I do the image, the network needs to accept high-dimensional input, and in MATLAB, the operation of the vector is much faster than for training. So I wrote a version of it that could accept vector input and vector output through a BP algorithm to supervise training. The BP algorithm can be referenced here: Backpropagationalgorithm, the main calculation of each layer of each node residuals on it. In addition, my code can be checked by gradient, but in some training sets, the value of the cost function will rise with the number of iterations, which is very strange, and then reduce the learning rate or the same. But in some simple points of the training set can still work, although the training error is also very large (not fully fit training samples). So if you find that the code inside the wrong part, but also hope you tell the next.

The main code is shown below:

learnrbf.m

[CPP] view plain copy

2. %//This was a RBF network trained by BP algorithm

4. %//Author:zouxy

6. %//date:2013-10-28

8. %//homepage:http://blog.csdn.net/zouxy09

10. %//email: [Email protected]

14. Close all; Clear CLC

18. %%% ************************************************

20. %%% ************ Step 0:load data ****************

22. Display (' Step 0:load data ... ');

24. % train_x = [1 2 3 4 5 6 7 8]; % each sample arranged as a column of train_x

26. % train_y = 2 * train_x;

28. train_x = rand (5, 10);

30. train_y = 2 * train_x;

32. test_x = train_x;

34. test_y = train_y;

38. Percent from Matlab

40. % RBF = NEWRB (train_x, train_y);

42. % output = RBF (test_x);

48. %%% ************************************************

50. %%% ******** Step 1:initialize parameters ********

52. Display (' step 1:initialize parameters ... ');

54. NumSamples = Size (train_x, 2);

56. rbf.inputsize = Size (train_x, 1);

58. Rbf.hiddensize = NumSamples; % num of Radial Basis function

60. rbf.outputsize = Size (train_y, 1);

62. Rbf.alpha = 0.1; % learning rate (should not being large!)

66. Percent Centre of RBF

68. For i = 1:rbf.hiddensize

70. % randomly pick up some samples to initialize centres of RBF

72. index = Randi ([1, numsamples]);

74. Rbf.center (:, i) = train_x (:, index);

76. End

80. Percent Delta of RBF

82. Rbf.delta = rand (1, rbf.hiddensize);

86. Percent Weight of RBF

88. R = 1.0; % random number between [-R, R]

90. Rbf.weight = rand (rbf.outputsize, rbf.hiddensize) * 2 * R-R;

96. %%% ************************************************

98. %%% ************ Step 2:start Training ************

100. Display (' Step 2:start training ... ');

102. Maxiter = 400;

104. Precost = 0;

106. For i = 1:maxiter

108. fprintf (1, ' iteration%d, ', i);

110. RBF = TRAINRBF (RBF, train_x, train_y);

112. fprintf (1, ' The cost is %d \ n ', rbf.cost);

116. Curcost = Rbf.cost;

118. if ABS (Curcost-precost) < 1e-8

120. Disp (' reached iteration termination condition and termination now! ');

122. Break ;

124. End

126. Precost = Curcost;

128. End

134. %%% ************************************************

136. %%% ************ Step 3:start testing ************

138. Display (' Step 3:start testing ... ');

140. Green = Zeros (rbf.hiddensize, 1);

142. For i = 1:size (test_x, 2)

144. for j = 1:rbf.hiddensize

146. Green (j, 1) = Green (test_x (:, i), Rbf.center (:, J), Rbf.delta (j));

148. End

150. Output (:, i) = Rbf.weight * Green;

152. End

154. Disp (test_y);

156. Disp (output);

trainrbf.m

[CPP] view plain copy

2. function [RBF] = TRAINRBF (RBF, train_x, train_y)

4. %%% Step 1:calculate Gradient

6. NumSamples = Size (train_x, 2);

8. Green = Zeros (rbf.hiddensize, 1);

10. Output = Zeros (rbf.outputsize, 1);

12. Delta_weight = Zeros (rbf.outputsize, rbf.hiddensize);

14. Delta_center = Zeros (rbf.inputsize, rbf.hiddensize);

16. Delta_delta = Zeros (1, rbf.hiddensize);

18. Rbf.cost = 0;

20. for i = 1:numsamples

22. Percent Feed forward

24. for j = 1:rbf.hiddensize

26. Green (j, 1) = Green (train_x (:, i), Rbf.center (:, J), Rbf.delta (j));

28. End

30. Output = Rbf.weight * Green;

34. Percent back propagation

36. DELTA3 =-(train_y (:, i)-output);

38. Rbf.cost = rbf.cost + sum (delta3.^2);

40. Delta_weight = delta_weight + delta3 * Green ';

42. DELTA2 = Rbf.weight ' * delta3. * GREEN;

44. for j = 1:rbf.hiddensize

46. Delta_center (:, j) = Delta_center (:, J) + Delta2 (j). * (train_x (:, i)-Rbf.center (:, J))./Rbf.delta (j) ^2;

48. Delta_delta (j) = Delta_delta (j) + DELTA2 (j) * SUM ((train_x (:, i)-Rbf.center (:, J)). ^2)./Rbf.delta (j) ^3;

50. End

52. End

56. %%% Step 2:update Parameters

58. Rbf.cost = 0.5 * rbf.cost./numsamples;

60. Rbf.weight = rbf.weight-rbf.alpha. * delta_weight./numsamples;

62. Rbf.center = rbf.center-rbf.alpha. * delta_center./numsamples;

64. Rbf.delta = rbf.delta-rbf.alpha. * Delta_delta./numsamples;

66. End

Green.m

[plain] view plain copy

1. function Greenvalue = green (x, C, Delta)

2. Greenvalue = exp ( -1.0 * SUM ((X-C). ^2)/(2 * delta^2));

3. End

Five, code testing

First, I tested one-dimensional input, and the function to fit is simple, y=2x.

train_x = [1 2 3 4 5 6 7 8];

train_y = 2 * train_x;

So the expected output is:

2 4 6 8 10 12 14 16

My Code training iterations 200 times after the network output is:

2.0042 4.0239 5.9250 8.0214 10.0692 11.9351 14.0179 15.9958

The output of the MATLAB NEWRB is:

2.0000 4.0000 6.0000 8.0000 10.0000 12.0000 14.0000 16.0000

As you can see, MATLAB is perfect to fit AH. My one or the mean square error is still quite large.

Then I tested the high-dimensional input, and the training sample was obtained by using the MATLAB rand (5, 10), which generated a random number between 5 rows and 10 columns [0 1]. That means our sample is 10, and the dimensions of each sample are 5 dimensions. We are also testing a very simple function y=2x. The results are as follows:

I'm not going to say anything about the result. Expect everyone to find the code inside the wrong place, and then tell under, thank you very much.

The above are copied over, Bo master provides MATLAB code can also improve, but it is a good learning article.

Radial basis networks (RBF network)