Deep learning BP algorithm backpropagation and detailed example analysis

The inverse propagation algorithm is an important algorithm in the training of multilayer neural networks, and the principle and derivation process of directional propagation algorithm are emphatically explained in this paper. Therefore, for some basic neural network knowledge, this article does not introduce. Before understanding the inverse propagation algorithm, we must first understand the Feedforward neural network algorithm in the neural network. feedforward Neural Networks

The following diagram is a simple schematic diagram of a multilayer neural network:

Given a feedforward neural network, we use the following notation to describe the network:
L: The number of layers representing the neural network;
NL: Indicates the number of neurons in the L layer;
FL (∙): Represents the activation function of the L-layer neurons;
Wl∈rnlxnl−1: The weight matrix representing the l−1 layer to the L layer;
BL∈RNL: Indicates the bias of the l−1 layer to the L layer;
ZL∈RNL: Indicates the input of the neuron in the L layer;
AL∈RNL: Indicates the output of the neuron of the L-layer;

Feedforward Neural networks disseminate information through the following formula:
ZL=WL⋅AL−1+BLAL=FL (ZL) the above two formulas can be combined in the form of the following:
ZL=WL⋅FL (zl−1) +BL this through a layer of information transmission, you can get the last output of the network Y is:
X=a0→z1→a1→z1→⋯→al−1→zl→al=y

Inverse propagation Algorithm

After understanding the structure of feedforward neural networks, we have a feedforward neural network based on the information transfer process, which pushes to the inverse propagation algorithm. First of all, to be clear, the inverse propagation algorithm is to better and faster training feedforward neural network, the weight of each layer of neural network parameters and bias parameters.
Before deducing the theory of reverse propagation, first look at a graph that can visually reflect the reverse propagation process, which is drawn from the principles of training multi-layer neural network using the back propagation.

Principles of training multi-layer neural network using backpropagation

The project describes teaching process of multi-layer neural network employing backpropagation algorithm. To illustrate this process, the three layer neural network with the inputs and one Output,which are shown in the picture Bel OW, is used: Each neuron is composed of units. First unit adds products of weights coefficients and input signals. The second unit realise nonlinear function, called neuron activation function. Signal e is adder output Signal, and y = f (e) is output Signal of nonlinear element. Signal y is also output Signal of neuron. To teach the neural network we need training data set. The training data set consists of input signals (x1 and x2) assigned with corresponding target (desired output) Z. The network training is an iterative process. In each iteration weights coefficients of nodes is modified using new data from training data set. Modification is calculated using algorithm described Below:each teaching step starts with forcing both input signals from Training set. After this stage, we can determine output signals values for each neuron in each network layer. Pictures below illustrate how signal is propagating through the network, Symbols W (XM) n represent weights of connections B Etween Network input XM and neuron n in input layer. Symbols yn represents output signal of neuron n. Propagation of signals through the hidden layer. Symbols wmn represent weights of connections between output of neuron m and input of neuron n in the next layer. Propagation of signals through the output layer. In the next algorithm step the output signal of the network y are compared with the desired output value (the target), whic H is found in training data set. The difference is called error signal D of output layer neuron. It is impossible to compute error signal for internal neurons directly, because output values of these neurons are unknown . For many years the effective method for training multiplayer networks have been unknown. Only the middle Eighties the backpropagation algorithm have been worked out. The idea was to propagate error signal D (computed in a teaching step) back to all neurons, which output signals were Input for discussed neuron. The weights ' coefficients wmn used to propagate errors back is equal to this used during computing output value. The direction of data flow is changed (signals was propagated from output to inputs one after the other). This technique are used for all network layers. If propagated errors came from few neurons they is added. The illustration is below: When the error signal for each neuron are computed, the weights coefficients of each neuron input node may modified. In formulas below DF (e)/de represents derivative of neuron activation function (which weights is modified). Coefficient h<