Caltech Open Course: machine learning and Data Mining _ Neural Network (Lesson 10)

Course introduction:

This section mainly introduces the artificial neural network. by introducing the evaluation model, the stochastic gradient descent method, the biological inspiration and sensor system, the lecturer explained the running process and principle of the neural network in detail using the view and mathematical analysis.

Course outline:

1. Stochastic Gradient Descent (random gradient descent method)

2. Neural Network Model)

3. Backpropagation Algorithm (reverse Propagation Algorithm)

4. Summarize

1. Stochastic Gradient Descent

In the previous lesson, we used the gradient descent method to find the optimal direction, and then moved a small step in this direction, but it is likely to cause a poor local optimal solution, in addition, all vertices need to be calculated for each move, so the computing volume is large. Then, if we only look at one point at a time and find the optimal direction for moving based on the current point, when all the points are selected, the general direction of moving is theoretically consistent with the previous method, because the mathematical expectation is equal. However, the random selection of a point may allow us to escape the most advantageous trap of the very poor part. Of course, we cannot escape all the traps. That's because when you select a point at will, the direction is not necessarily the direction of optimization. Therefore, the next step may be to move to the high point, and then find a new way out in the new position, in this way, there is a greater possibility to obtain better local advantages. Consider the following image. In the left figure, when the initial position we selected for the first time is the first black spot, then according to the overall gradient descent method, it would be a bit difficult to get into that very bad local mouth. However, if we choose the random gradient descent method, it is possible that the direction of a point is deviated from the most advantageous, so that the forward direction is to the right, in this way, you can skip this trap. According to the gradient descent method in the figure, when a gentle area is encountered, the method considers that the best advantage is found, and it will stagnate. However, the random gradient descent method pushes the algorithm forward due to inconsistent directions of different points, so we escaped this trap again.

The following benefits apply to the random gradient descent method:

1. Low computing cost, because only one vertex is selected at a time for calculation.

2. Randomness helps to escape the local minimum point and find a better minimum point.

3. Simple because only one vertex is selected for calculation at a time

2. Neural Network Model)

When learning the sensor model, we know that when there are more than three input data points, we cannot correctly classify all the points in the worst case, that is, DVC = 3. However, if we use two sensor models for classification and then combine the results of these two models to obtain the most general results, we can classify all four points, for example:

For more points, we can use more models for classification. By combining the results of different models, it is possible to find the optimal solution (weight) through the sensor model ).

Use Form for intuitive explanation:

There are five sensor models (each circle represents a sensor model, also called a neuron or node ). The sensor outputs the final result.

Each line has some numeric values, which are the weights (parameters) We need to learn. They are unknown at the beginning. The above model is actually a neural network.

Theoretically, any output can be produced through the arrangement and combination of these sensors (any.

Definition:

The neural network model is a comprehensive model obtained by combining different simple models. The combination method is shown in:

The first column is the input data, the rightmost is the output, and the rest is the hidden layer. The layer of the network is L = 3.

Each line represents a weight and needs to be learned and determined. Each θ represents a model (any model can be different from each other ), each line belongs to the same level as the model it points.

The input of the next model is the output of the previous model multiplied by the corresponding weight (the value of the line ).

To facilitate the discussion of neural networks, assume that θ (S) = (1-e ^ s)/(1 + e ^ s) of each model, and the processed data is binary classification.

In addition, the following symbols are defined:

W is the parameter we need to learn. corresponding to the line in the figure, the upper Mark L indicates that the current W belongs to the L layer, and the subscript indicates the node location starting from the line. J indicates the node to which the line points. It is worth noting that the value range of I can be obtained because there is a constant term: x0.

So we have:

X indicates output, and s indicates input.

When l = 0, it indicates the original data. At this time, 0 <j <D + 1

Solving X is a recursive process. First layer 1st is obtained, and then Layer 2 is obtained until the final output.

This model solves the following two problems:

1. generalization: a large number of models, parameters, weights, and so on lead to a great degree of freedom. As mentioned in the error Deviation Analysis, assuming a large set increases the difficulty of finding the optimal function.

2. Optimization: how can we learn so many parameters well?

The third part will try to solve the above problems.

3. Backpropagation Algorithm (reverse Propagation Algorithm)

This part is basically a mathematical analysis. It looks a little dizzy and doesn't understand very well. It can only be a constant...

The random gradient descent method only pays attention to one point at a time. Therefore, we need to find the partial differentiation of Ein (w) on each data: ▽ E (W ). E (w) = (H (Xn), yn ). For convenience, make E (w) = (H (Xn)-yn) ^ 2 (theoretically, all valid error measurement functions can be used ).

Yes =

Because SJ (L) = xi L-1) * wij (l), (note: Here (l) indicates the upper mark as l): order,

When l = 1, XI (l) indicates the original input data. Therefore, we can recursively obtain the second item on the right of the preceding equation, as long as we can find the first item, we can learn W.

To obtain the first item, start from the last layer, because the last layer is closest to the output of the entire algorithm, and θ (S1 (L) is x1 (L). (Note: where l is the superscript), there are:

After the last layer is obtained, the preceding values can be obtained recursively. The requirements and requirements are as follows: (the node at the L-1 layer is fed back by all nodes at the L layer)

The following is the pseudo code:

Summary:

The first part undertakes the previous lesson and introduces a better gradient measurement method, which also lays the foundation for learning neural network parameters. The second part mainly introduces the basic concepts and functions of neural networks. The last part is the mathematical analysis, which converts the neural network model to the computer processing level. Therefore, the last part is the theory to practice.

Caltech Open Course: machine learning and Data Mining _ Neural Network (Lesson 10)