Derivation of BP algorithm formula

first, to understand the basic definition,

Such as

where n represents the number of training samples, vector x(n) represents the nth sample input data at the input layer, and vector D(n) represents the desired output at the output layer.

If the Neuron j is located in the output layer of the network, the error generated by the neuron output is defined As:

Where the desired output is expressed, representing the actual output, indicating the error generated by the Neuron J Output.

The instantaneous error energy generated by the Neuron J is defined As:

Then the total instantaneous error energy generated by the entire network is:

where c represents the set of neurons in the network output layer.

The induction function at the activation function of the neuron J (induced Local Domain) is:

where m is the input number of neuron j, indicating the synaptic weight from neuron I to neuron j, is the bias of neuron j, corresponding to the function output signal of neuron I connected to Neuron J.

The function signal of the neuron j, i.e. the output is:

wherein, the activation function of the corresponding neuron J is Expressed. (for simplicity, the activation function for all neurons in the network is usually the same function).

The inverse propagation of BP algorithm refers to a correction value when the weight adjustment is proportional to the partial derivative, according to the chain rules of differential can be written as,

The derivative of equation 4 can be obtained by:

(the result is not such a result because it is biased against a neuron j)

The derivative of Equation 2 can be obtained by:

The derivative of Equation 6 can be obtained by:

The derivative of equation 5 can be obtained by:

Take equation 8, 9, 10, 11 into Equation 7,

The correction of weights is defined by the delta rule as:

The learning rate parameter, which is a constant, can be specified; the minus sign indicates the direction in the weight space to change the weight of the descent (even if the overall error energy of the network is reduced).

Take equation 12 into equation 13,

Where is the local gradient, defined As:

It can be seen that the local gradient of the neuron J equals the multiplication of the derivative of the Neuron's error signal and its activation function to the induced partial domain.

So get,

which can be specified in the BP algorithm forward to the calculation process can be obtained, the activation function is known, through the calculation can be obtained, focusing on the Calculation. It is clear that the location of the neuron J needs to be discussed in Categories:

1. The Neuron j is located in the output layer, which is the simplest case where the expected output and actual output of the neuron are known (the actual output is derived from the forward calculation of the algorithm),

Thus the synaptic weights of neuron I to nerve J can be adjusted.

2. Neuron j is not located in the output layer, but in the hidden layer. At this point the error signal can no longer be determined according to equation 2, because it is unknowable. The error signal needs to be determined back to the counter by the error signal of the neuron directly connected to it. The neuron at the output layer connected to the Neuron J is k, which needs to be redefined,

The emphasis is on the calculation,

For equation 17 to be differential,

(because It is biased, and each one is a function, all contains), the same is true with chain rules,

And because

So

According to the previous solution, you can know that

where M is the number of neurons in the network layer where the neuron J Resides. For equation 22 to be differential,

You can get the equation 21 and 23 into equation 19,

Take equation 24 into equation 16 to get,

The equation 25 and Equation 14 can be obtained,

This allows the weights of neuron I to neuron j to be adjusted.

To sum up, you can know that the weight of the adjustment core is that,

Derivation of BP algorithm formula