BP Neural network

bp neural network in BP for back propagation shorthand, the earliest it was by Rumelhart, McCelland and other scientists in 1986, Rumelhart and in nature published a very famous article "Learning R Epresentations by back-propagating errors ". With the migration of the Times, the theory of BP neural network has been improved and updated, which has undoubtedly become one of the most widely used neural network models. Let's explore the initial basic models and concepts of the BP neural network!

Speaking from the biological model of neural network

We know that the transmission of human brain information, the response to external stimuli are controlled by neurons, the human brain is composed of such neurons in the upper Bai. These neurons are not isolated and closely linked, each neuron is connected to thousands of neurons on average, thus constituting the neural network of the human brain. Stimulation in neural networks is followed by certain rules, and a neuron does not respond to stimuli that are transmitted from other nerves every time. It will first accumulate the stimulation of its neighboring neurons, and at some point generate its own stimulation to pass it on to some neurons adjacent to it. The Bai neurons in this work form the brain's response to the outside world. The mechanism by which the human brain learns to stimulate the outside world is by regulating the connections between these neurons and their intensity. Of course, what is actually said is a simplified biological model of the real neural work of the human brain, which can be generalized to machine learning using this simplified biological model and described as an artificial neural network. BP Neural network is one of them, to look at the specific analysis of neurons.

Fig. 1 Neuron in neural network

The stimulation of neuronal accumulation is the sum of the stimuli transmitted by other neurons and the corresponding weights, which are expressed in Xj , andYi denotes the amount of stimulation transmitted by a neuron,Wi Represents the weight of a link to a neuron stimulation, resulting in a formula:

Xj = (y1 * W1) + (y2 * W2) +...+ (Yi * Wi) +...+ (yn * Wn)

And when Xj completes accumulating, the accumulated neurons themselves are stimulated by the propagation of some neurons around them, which are expressed as YJ to be shown below:

YJ = f (Xj)

The neurons were treated by the results of the accumulated Xj , and the YJ was stimulated externally. This processing is represented by an f function map, which is called an activation function .

The composition of BP neural network

After analyzing the individual neurons, and then looking at the situation after they make up the network, it is the most intuitive way to illustrate them graphically, as shown in 2:

Figure 2 BP Neural network

In the first region, they are equivalent to external stimuli, are sources of stimulation and transmit stimuli to neurons, so the first region is named the input layer . The second area, which indicates that neurons transmit stimuli to each other is equivalent to the inside of the brain, so the second area is named the hidden layer . The third region, which expresses the response of neurons to the outside world after multi-level transmission, is named the output layer of the third region.

The simple description is that the input layer passes the stimulus to the hidden layer, and the hidden layer passes the stimulus to the output layer through the intensity (weight) of the link between the neurons and the transfer rule (activation function), and the output layer organizes the hidden layer processing after the stimulation produces the final result. If there is a correct result, then the correct result and the resulting results are compared, get the error, and then inverse push the link weights in the neural network feedback correction, so as to complete the learning process. This is the BP neural network feedback mechanism, is also the source of the BP (back propagation) name: The use of backward feedback learning mechanism to correct the weight in the neural network, and finally achieve the goal of output correct results!

Mathematical derivation of BP neural network

Based on the mathematical analysis of BP Neural network model, the first part of the neural network model in this paper can obtain a BP neural network formula (1):

For the activation function of the output of the neuron itself, generally choose the Sigmoid function , then you can get the second formula (2):

Through the above two formulas, we can analyze the calculation process of output results in BP neural network. Each neuron receives the stimulus Yi and then the weighted accumulation (weighted Wji ) is completed after the XJ is produced, and then stimulated by the activation function of yJ , The next layer of neurons to which it is connected is passed, and, in turn, results are eventually output.

We will then analyze how to use the backward feedback mechanism to correct the neuron weight Wji, this part of the mathematical derivation needs to apply to the mathematical content of the multivariate differential. To fix the Wji , you need to get the amount of error. In particular, the first to use a DJ to express the true correct results, and set the error to e , then (YJ-DJ) corresponds to the e for yj differential increment, that is yJ minus (YJ-DJ) can get the correct value, get the formula (3):

Then, to clarify the target, you need to know what is the amount of error for the weight Wji is also the value . In the equation (1) It is known that the Wji is related to XJ , then the formula (4)can be deduced:

The error amount of Wji needs to be obtained and converted to the required value. It is deduced as follows:

The value of which can be derived from the formula (2) :

So the value of the resulting error amount is:

The above formula needs to pay attention to subscript: The last one is Yi , the front is YJ . Pushing here can be a complete use of the neural network output value YJ and the correct value of the DJ to the last layer of hidden layer Wji correction, then the other hidden layer? Then look down.

The above derivation process starts with the formula (3) , if we know (note is Yi , the formula (3) is YJ ), it is possible to deduce the same with other hidden layers to find out the weight value of the error amount of correction. The derivation is as follows:

So all the error amount can be the same deduction to complete!

The last step of correcting Wji is to add the following variables, setting an L (0 to 1) learning rate.

At this point, the BP neural network feedback part of the mathematical deduction is completed, you can draw on their own draft paper ~

Reprint: http://www.cnblogs.com/jzhlin/archive/2012/07/28/bp.html

BP Neural network