Machine Learning Week 8th-smelting number into gold-neural network

Reference books

Deep learning

Deep learning is a new field in machine learning research, and its motive is to establish and simulate the neural network of human brain import analysis and learning, which imitates the mechanism of human brain to interpret the data.

Examples of images, sounds and text. Deep Learning is a kind of unsupervised learning.
The concept of deep learning is derived from the research of artificial neural network. Multilayer perceptron with multiple hidden layers is a kind of deep learning structure.

Deep learning combines low-level features to form more abstract higher-layer representations of attribute class-ring features to discover distributed feature representations of data.
The concept of deep learning was proposed by Hinton and others in 2006. Based on the belief that the network (DBN) proposes an unsupervised greedy-to- layer training algorithm , it brings hope for solving the optimization problems related to deep structure, and then puts forward multilayer automatic encoder deep structure.

In addition, the convolution neural network proposed by LeCun is the first real multi-layer structure learning algorithm, which uses spatial relative relationship to reduce the number of parameters to improve the training performance.

BP Neural network

Error back propagation, base error inverse propagation algorithm
1986 by team of scientists led by Rumelhart and McCelland
Multilayer Feedforward Network
One of the most widely used neural network models at present
multi-layered, can be adjusted a lot of weights , you can use a variety of activation functions
Can learn and store a large number of input-output pattern mapping relationships without having to reveal the mathematical obnoxious that describe this mapping relationship beforehand
The learning rule uses the steepest descent method to continuously adjust the weights and thresholds of the network by reverse propagation , so that the error level of the network is obnoxious and minimized.
The BP neural network model topology consists of input layers (inputs), hidden layers (hide layer), and output layers (outputs layer)

Multilayer Feedforward Neural Network

Hidden layers do not hide nodes
Feedforward-nodes in each layer are connected only to the next layer of nodes

Learning rules: The steepest descent BP method

The idea of multilayer neural networks is very early, but there is no suitable learning algorithm
The algorithm of error reverse propagation is used to ripen artificial neural network
Basic idea: The learning process consists of two processes, the forward propagation of signals and the reverse propagation of errors.
"Artificial neural network theory, Design and Application" page 47th

Realization of image compression by BP neural network

The BP neural network realizes image compression. Data compression based on nonlinear mapping capability of BP network
JPEG compression
Image compression algorithm based on wavelet transform
Fractal Compression coding
Vector quantization compression coding
Using BP neural network to compress the gray image, a high compression ratio is achieved under the condition of good peak signal-to-noise ratio (PSNR).

Common file compression software such as WinZip, WinRAR and so on is lossless compression, can fully restore the original file content. Multimedia information has the characteristics of large amount of information and redundancy, often using lossy compression technology.

1. According to the principle of large area coloring, the image must be in a certain area of memory in the same or similar color, for the observation of the human eye only makes sense, otherwise see only the chaotic snowflakes.

Therefore, there is similarity between the neighboring pixels in the image, which results in the prediction coding of the image.

2. Because of the visual masking effect, the human eye is often not sensitive to color detail.

The small loss of image information is often not perceived or acceptable, which provides a wide range of compressed space 3. There is statistical redundancy in the data,

In an image depicting the ocean, the blue color may appear much higher than the red color, and compression can also be achieved by removing statistical redundancy.

BP neural Network for compression:
Only one hidden layer is used, so the whole constitutes a three-layer network.
A set of input patterns is mapped to a set of output modes through a small number of hidden layer elements, and the output mode is as equal as the input mode.

Therefore, the value of the hidden layer neuron and the corresponding weight vector can output a vector that is the same as the original input pattern.

When the number of neurons in the hidden layer is small, it means that the hidden layer can represent the input pattern with fewer numbers, which is actually compression.

The first layer is the input layer, the middle layer is the hidden layer, and the network mapping function relies on the hidden layer implementation. The transformation of the input layer to the hidden layer is equivalent to the coding process of compression, while the transformation from the hidden layer to the output layer is equivalent to the decoding process

Both the input layer and the output layer are composed of m neurons, and the hidden layer contains k neurons. The network compresses the data, stores it in the hidden layer, and then outputs it to the next layer from the hidden layer.

Using the feedforwardnet function of MATLAB Neural Network Toolbox, the BP network is created and the training algorithm is specified. In order to achieve good results, the LM training method was used.
Save the results. After the training is completed, the result of compression is the value of the hidden layer neuron vector corresponding to each input mode, as well as the weights and thresholds of the network. Save As Mat file using the Save command

Steps:

Extract.

To load a data file using the load command

Inverse Normalization of data

Reconstruction

Inverse normalization of images

Image Block Recovery

Run BP_IMAGECOMPRESS.M for compression

Bp_imagerecon for decompression

Data Source: ftp://ftp.ics.uci.edu/pub/machine-learning-databases

Credit evaluation of personal credit based on BP network

German Credit Database

The evaluation method of personal credit is divided into two kinds, qualitative evaluation and quantitative evaluation,

The former is mainly based on the subjective judgment of the credit personnel, the latter is based on the personal customer data, using scorecard and credit scoring model tools for analysis.

This example uses the BP neural network, takes the known user information and the credit condition as the training sample, studies obtains an abstract model, then evaluates the new sample,

The correct rate of stability in more than 70% of all customers do two classification, only distinguish between good and bad two cases. The data is based on the German credit database.

The German credit database is organized by Hans Hofmann, which contains 1000 customer profiles, each with 20 attributes and a good or bad credit rating.

(1) Read in the data.

(2) Divide training samples and test samples. Of the 1000 samples, there were 700 positive cases (good reputation) and 300 negative cases (poor reputation).

The first 350 positive cases and the first 150 negative examples were taken as training samples, 350 positive cases and 150 negative examples were taken as test samples.

(3) Normalization of samples. Normalization of input samples using the Mapminmax function

(4) Create a BP neural network and complete the training

(5) test. The BP network output value is not limited to 1 or 2, but is a real number, so you also need to convert the output to an integer.

1.5 is the threshold value, the output less than this threshold is sentenced to 1 (good credit), or 2 (poor credit)

Run script CREDIT_CLASS.M Test 20 times the average correct rate is 74.97%, the lowest correct rate is 73.4%, the number of iterations is 3 times

Radial basis function RBF

Norm
What are radial basis functions
Commonly used radial basis functions: Gaussian function, inverse s-type function, quasi-two-time function
Approximation interpretation
"Artificial neural network theory, Design and Application" page 165th

Regularization of RBF Neural network

The number of hidden layer nodes is the number of samples
Use the Gaussian function as a hidden layer activation function called the base function
The data center is a sample point, and the extension constants can be determined by the sample distribution (the distance between 22) and the number of samples. The base function of all nodes takes the same extended constant
The output layer uses a linear activation function
is a universal approximation device
Learning algorithms, like LMS
"Artificial neural network theory, Design and Application" page 167th

Comparison between RBF and BP

RBF has only one hidden layer, BP can be multilayer

The RBF hidden layer and output layer use different activation functions , BP is the same
The characteristics of the activation function are different, the RBF chooses the nonlinear function of local exponential attenuation (emphasizing local), and BP uses the input sample to take the inner product of the weight vector as the independent variable, and the activation function uses the sigmoid ring limiting function (emphasizing the global).
BP is the global approximation to nonlinear mappings, and the RBF is more prominent in local characteristics (recalling the SVM case)

Generalized RBF Neural Network

The basic idea is to map the low-dimensional linear irreducible problem (nonlinear problem) to the high-dimensional space to solve the problem of linear variational problems (radial basis function is used to approximate the mapping).
Number of hidden layer nodes general input less than sample number
The center is not necessarily at the sample point, through the training algorithm (may need clustering) to select
Extension constants are no longer uniform and are derived from training algorithms
The output function includes the threshold value in the linear function
Training algorithm (based on K-means clustering)

Machine Learning Week 8th-smelting number into gold-neural network