MATLAB Neural network Programming (v) Model structure and learning rules of--BP neural network

"Matlab Neural network Programming" Chemical Industry Press book notes
The fourth Chapter 4.3 BP propagation Network of forward type neural network

This article is "MATLAB Neural network Programming" book reading notes, which involves the source code, formulas, principles are from this book, if there is no understanding of the place please refer to the original book

The learning rules and LMS learning algorithms of Perceptron neural network can only train single-layer neural networks, while single-layer neural networks can only solve the problem of linear classification. Multilayer neural networks can be used for nonlinear classification problems, but it is necessary to find a learning algorithm for training multi-layer networks.

One, what is BP neural network

The BP (back propagation) network was presented by a team of scientists led by Rumelhart and McCelland in 1986 and is a multi-layered feedforward network trained on the error inverse propagation algorithm . is one of the most widely used neural network models. The BP network can learn and store a large number of input-output pattern mapping relationships without having to reveal the mathematical equations that describe the mapping relationship beforehand. Its learning rule is to use the steepest descent method, through the reverse propagation to continuously adjust the network weights and thresholds, so that the network error squared and minimum. The BP neural network model topology includes the input layer, the hidden layer, and the output layer. ”

1, "is a multi-layer Feedforward network trained by error inverse propagation algorithm"
BP is the English abbreviation for the post-propagation, so what is the object of transmission. What is the purpose of communication. The way to spread is back, but what does that mean?

The object of propagation is error, the purpose of propagation is to get the estimation error of all layers, and the latter is to deduce the front layer error by the back layer error:

That is, the idea of BP can be summed up

Using the error of output to estimate the error of the direct leading layer of the output layer, and then using this error to estimate the error of the previous layer, the error of all the other layers is obtained when the inverse of the layer is passed down.

2, "The BP Neural network model topology includes input layer (inputs), hidden layers (hide layer), and output layer"

A simple three-layer BP network:

BP algorithm is proposed to solve the weight coefficient optimization of multilayer feedforward neural networks; Therefore, the BP algorithm usually implies that the topological structure of neural network is a non-feedback multilayer feedforward network. therefore Sometimes also called the non-feedback multilayer Feedforward network is the BP model.

3, "The BP network can learn and store a large number of input-output pattern mapping relationships without having to reveal the mathematical equations describing this mapping relationship beforehand." ”

BP uses a function called activation to describe the relationship between layer and layer output, thus simulating the interaction between the neurons in each layer.

The activation function must satisfy conditions that are everywhere. Then the more commonly used is an activation function called the S-type function:

Of course, other activation functions will be used, which will be explained in detail below.

4,
Neural Networks for learning purposes:

I want to be able to learn a model that can output a desired output to the input.

The way to learn:
Changing the connection weights of the network under the stimulation of the external input sample

The nature of Learning:
Dynamic adjustment of each connection weight value
The core of learning:

Weight adjustment rules, that is, in the learning process, the connection rights of each neuron in the network changes based on a certain adjustment rules.

The above content references the document: http://www.cnblogs.com/wentingtu/archive/2012/06/05/2536425.html

The neural network trained by BP algorithm is called BP Neural network. In the time application of artificial neural network, BP Network is widely used in function approximation, pattern recognition and classification, data compression and so on. 80%-90% Artificial Neural network model is the BP network or its deformation, BP network is also the core of Feedforward neural network, reflecting the most important part of artificial neural network.

Two, BP network model structure

1, neuron model.
A basic BP neuron model is shown in the following diagram:

It has r inputs, each input is connected by an appropriate weight of W and the next layer, and the output of the network can be expressed as:

A=f (WP+B)

The transfer function of hidden layer neurons in BP network is usually pureline with log-sigmoid type function logsig,tan-sigmoid type function tansig and pure linear function.
If the output layer of the BP network is sigomoid type neuron, the output of the whole network is limited to a small range, and if it is a pureline linear neuron, the output of the whole network can be arbitrary.

The transfer function of BP network must be a monotonically increasing function . It is very important to calculate the reciprocal of function logsin,tansig,pureline in the course of BP network training, so the MATLAB Neural Network Toolbox provides direct function Dlogsig,dtansig,dpureline of these functions.

2, Feed-forward neural network structure.
The Feedforward neural network usually has one or more hidden layers, and the neurons in the hidden layer adopt the S-type transfer function (such as the logsig mentioned above), and the neurons in the output layer adopt the linear transfer function.
The nonlinear transfer function neurons of the hidden layer can learn the linear and nonlinear relations between the input and output, and the linear output layer is to widen the network output. If you need to limit the output of the network (as defined between 0 and 1), you can use the S-type transfer function.
The following figure is a typical two-layer BP network, the neuron transfer function of the hidden layer tansig, the output layer neuron transfer function pureline.

third, BP learning rules

The generation of BP network is attributed to the acquisition of BP algorithm. The BP algorithm belongs to the Δ algorithm and is a supervised learning algorithm. The supervised BP model training indicates that we have a training set that includes: input X and the output Y that it is expected to have; So for the current BP model, we can get the error of the training set.
BP algorithm consists of two parts: forward transmission of information and reverse propagation of error. in the process of forward propagation, the input information passes through the implicit layer to the output layer, and the state of the neurons in each layer only affects the state of the next layer. If the output layer does not get the desired output, then the error value of the output layer is computed and then transmitted in reverse, and the error signal is reversed along the original frontal connection path through the network to modify the weights of each layer neuron until the desired target is reached.

So the core idea of BP is: The output error in some form through the hidden layer to the input layer of the anti-pass, some of the form here is actually:

This is the process of "forward propagation of a signal and reverse propagation of errors":

The following is an example of the BP algorithm derivation, the simplified diagram of the network is as follows:

The BP algorithm needs to use the first derivative of the activation function of each layer, so its activation function is required to be differentiable everywhere.

The above process can be represented by a schematic diagram:

Summary: Flow of BP algorithm

1. Forward propagation

The input samples are processed from the input layer through a layer of hidden elements, then the output layer is passed through all the hidden layers, and the state of each layer of neurons is affected only by the state of the next layer of neurons. The output layer compares the current output to the desired output, and if the current output does not equal the desired output, it enters the reverse propagation process.

2. (Error) Reverse propagation

In reverse propagation, the error signal is transmitted back to the original forward propagating path, and the weights of each neuron in each hidden layer are modified to minimize the error signal.

Reference documents
Http://www.cnblogs.com/wengzilin/archive/2013/04/24/3041019.html
Http://www.cnblogs.com/wentingtu/archive/2012/06/05/2536425.html