Go Introduction and realization of BP artificial neural network

Neural network concepts and suitability fields

The earliest research of neural network was proposed by the 40 psychologist McCulloch and mathematician Pitts, and their MP model was the prelude of Neural Network research.

The development of neural networks has gone through 3 stages: 1947-1969 years early, during which time the scientists put forward many neuron models and learning rules, such as MP model, Hebb learning rules and perceptron, etc. the neural network control is at a low ebb from the end of the 60 to the 80 's, as is the case with the whole neural network. During this time, scientists have done a lot of work, such as Professor Hopfield on the network to introduce the concept of energy functions, the stability criteria of the network, proposed for associative memory and optimization of the method of calculation. In 1984, Professor Hiton put forward Bol Tzman machine model, 1986 Kumelhart and other people put forward the error back propagation neural network, referred to as BP network. At present, BP Network has become a widely used network. 1987 to present for the development period, in this period, the neural network has received international attention, each country has carried on the research, forms the neural network development another climax.

Artificial neural Network (ANN) is a kind of simulation and approximation of biological neural network, which simulates the biological neural network from the structure, the realization mechanism and the function. From a systematic point of view, artificial neural network is an adaptive nonlinear dynamic system composed of a large number of neurons through extremely rich and perfect connections. Artificial neural networks, because the learning system of organisms is an unusually complex network of interconnected neurons, each of which has a certain number of real inputs and produces a single real value output. 1960 Widrow and Hof pioneered the use of neural networks for automatic control research. Neural network with its unique structure and processing information methods, in many practical applications have achieved significant results, the main applications are as follows: Automatic control field, processing combinatorial optimization problems, pattern recognition, image processing, sensor signal processing, robot control, signal processing, health care, medical, economic, chemical industry, Fields of welding, geography, data mining, power systems, transportation, military, mining, agriculture and meteorology.

Basic structure of neural network

Artificial neural network is composed of neuron model, this information processing network composed of many neurons has the parallel distribution structure. Each neuron has a single output and can be connected to other neurons; there are many (multiple) output connection methods, each connecting method corresponds to a connection weight coefficient. The ANN can be regarded as a processing element of the processing unit PE (elements) as a node, with a weighted arc (chain) connected to each other by the direction of the graph. The information from other processing units (neurons) I is XI, and their interaction strength with this processing unit is wi,i=0,1,..., n-1, the internal threshold value of the processing unit is θ. So the input of this neuron is:

The output of the processing unit is:

, Xi is the input of the element I, and WI is the interconnection weight of the first element and the processing unit. F is called the excitation function (activation functions) or the action function. It determines the output of the node (neuron). The output is 1 or 0 depending on its input and is greater than or less than the internal threshold θ.

The neuron cells shown are composed of multiple input xi,i=1,2,..., N and an output y. The intermediate state consists of the right and the representation of the input signal, and the output is:

Training Network

Neural network structure is designed to complete, with input, output parameters, we need to train the network. The training of neural networks includes training of perceptron training, Delta rule training and inverse propagation algorithm, in which Perceptron training is the foundation.

Perceptron and Delta Training rules

The first step in understanding neural networks is to start with an abstract biological neural network, which is based on a unit called Perceptron. The perceptron calculates the linear combination of these inputs with a real value vector, and if the result is greater than a certain threshold, output 1, otherwise output-1, if x is from 1 to N, then the Perceptron calculates the formula as follows:

Each WI is a real number constant, or weight, used to determine the contribution of input XI to the perceptron output. In particular,-w0 is a threshold value.

Although when the training sample is linearly separable, the Perceptron rule can successfully find a weight vector, but if the sample is not linearly separable it will not converge, so people have devised another training rule to overcome this deficiency, which is called the Delta rule. The Perceptron training rule is based on the idea that the adjustment of the weight coefficient is determined by the difference equation expression of the target and the output. The delta rule is based on a gradient descent approach. This complex mathematical concept can be represented by a simple example. From a given point of view, the South path is steeper than the East one. Eastward is like falling off a cliff, but southward is down a slightly sloping slope, a steep hill to the west, and the north to the ground, as long as you stroll slowly. So what you're looking for is to reduce the steep sum to the smallest path in all paths that reach the flat. In the adjustment of the weight coefficient, the neural network will find a method to reduce the error to the minimum weight coefficient distribution. This part we do not do a detailed introduction, if necessary, we can refer to the relevant artificial intelligence books.

Inverse propagation algorithm

Artificial neural Network Learning provides a practical method for learning real values and vector-valued functions, which can be used for both continuous and discrete attributes. And it has good robustness to the noise in the training data. The inverse propagation algorithm is the most common network learning algorithm. This is a common method that we know is used to train neural networks, and the inverse propagation algorithm is a highly learning system with simple structure and easy programming.

Rummelhat (Rumelhart) and McClelland (Meclelland) developed BP Network Learning algorithm in 1985, and realized the idea of multi-layer network of Minsky. BP network contains not only input nodes and output nodes, but also one or more layers of hidden (layer) nodes. The input signal is passed forward to the hidden node, and after the function, the output information of the hidden node is passed to the output node, and the output result is given finally. The S-type function is generally used for the excitation function of nodes.

The inverse propagation (BACK-PROPAGATION,BP) algorithm is a simple method to calculate the change of the network performance caused by the change of the individual weight value. Since the BP algorithm process consists of starting from the output node, it propagates backward to the first hidden layer (that is, the hidden layer closest to the input layer) by the total error caused by the weight correction, so called "reverse propagation." The inverse propagation characteristic is closely related to the nature of the problem solved and the choice of detail.

The inverse propagation algorithm can be used to learn the weights of this multilayer network for multi-layer networks formed by a series of determined cell interconnects. It uses the gradient descent method to try to minimize the squared error between the network output value and the target value, because we want to consider the network of multiple output units instead of just a single unit, so we re-compute the error E to sum up the error of all network outputs:

The Outpus is a collection of network output units, and TKD and OKD are related output values to the training sample D and K output units.

A fascinating feature of the inverse propagation algorithm is that it can find useful intermediate representations within the hidden layers of the network:

1. The training sample contains only the network inputs and outputs, and the weight adjustment process is free to set weights to define any hidden unit representations, which represent the most effective when error E is minimized.

2. Guide the reverse propagation algorithm to define new hidden layer features, which are not explicitly represented in the input, but can capture the characteristics most relevant to the learning objective function in the input instance

The inverse propagation algorithm for training neurons is as follows:

C + + Simple implementation and testing

The following C + + code implements the BP network, through 8 3-bit binary samples corresponding to an expected output, training BP network, the last trained network can be the input three binary number corresponding to the output of a decimal number.

View Code

Reference documents

Introduction to neural networks--a model learning http://www.ibm.com/developerworks/cn/linux/other/l-neural/index.html using the inverse propagation algorithm

Ai Java Tank Robot Series: Neural Network, Upper http://www.ibm.com/developerworks/cn/java/j-lo-robocode3/index.html

Ai Java Tank Robot Series: Neural network, Lower http://www.ibm.com/developerworks/cn/java/j-lo-robocode4/

Using Python to construct a neural network--hopfield network can reconstruct distorted patterns and eliminate noise http://www.ibm.com/developerworks/cn/linux/l-neurnet/

Provides the basic data of a MATLAB BP neural network http://www.cnblogs.com/galaxyprince/archive/2010/12/20/1911157.html

An implementation of CodeProject, the author has given several forms of application examples http://www.codeproject.com/KB/recipes/aforge_neuro.aspx

Go Introduction and realization of BP artificial neural network