An overview of BP neural network algorithm and practice

Neural networks have been very hot, there has been a period of depression, now because of deep learning reasons to continue to fire up. There are many kinds of neural networks: forward transmission network, reverse transmission network, recurrent neural network, convolution neural network and so on. This paper introduces the basic Reverse Transmission neural Network (backpropagation), mainly describes the basic flow of the algorithm and its own experience in training BP neural network.

The structure p> neural network of BP Neural network is the way of simulating the neural unit of human brain, but a great simplification, the neural network is composed of many neural network layer, and each layer is composed of many units, the first layer is called input layer, the last layer is called the output layer, the middle layer is called hidden layer , in the BP neural network, there is only a connection between each cell of the adjacent nerve layer, except the output layer, each layer has a bias node:

Although the hidden layer in the figure is only one layer, but the number of layers is not limited, the traditional neural network learning experience that the first layer is good enough, and the recent in-depth study does not think so. The bias node is to describe the characteristics of the training data, the bias node for each node in the next layer of different weights to produce different offsets, it can be considered that the bias is each node (in addition to the input layer) of the attributes. Our bias node is omitted from the diagram:

Before describing the training of BP neural networks, let's look at the properties of each layer of the neural network:

Each neuron unit has a certain amount of energy, we define its energy value as the output value of the node J OJ;

The connection between the adjacent layers has a weighted wij, whose value is between [ -1,1];

In addition to the input layer, each node of each layer has an input value, which is the sum of the energies of all the nodes in the upper layer, which is transmitted by weight, plus the bias;

In addition to the input layer, each layer has a bias value between [0,1];

In addition to the input layer, the output value of each node is a nonlinear transformation of the input value of the node;

We think that the input layer has no input value, its output value is the training data attributes, such as a record x=< (1,2,3), class 1>, then the input layer of the three nodes output values are 1,2,3 respectively. Therefore, the number of nodes in the input layer is generally equal to the number of attributes of the training data.

Training a BP neural network, in fact, is to adjust the weight of the network and offset these two parameters, the BP neural network training process is divided into two parts:

Forward transmission, passing output value by layer wave;

Reverse feedback, adjust the weight and bias in reverse layer;

Let's look at the forward transmission first.

Forward transfer (Feed-forward forward feedback)

Before training the network, we need to randomly initialize weights and biases, take a random real number [ -1,1] for each weight, each offset a random real number of [0,1], and then start forward transmission.

Neural network training is done by multiple iterations, each iteration using all the records of the training set, and each training network using only one record, the abstract description is as follows:

First set the output value of the input layer, assuming that the number of attributes is 100, then we set the input layer of the number of neurons is 100, the input layer of the node NI is recorded on the first dimension of the attribute value XI. The operation of the input layer is so simple, after each layer will be more complicated, in addition to the input layer, the other layers of input value is the upper layer of input value by weight added to the result value plus bias, each node output value, such as the input value of the node to transform