Artificial neural Network (Artificial neural netwroks) Notes-basic non-deterministic statistical training algorithms

In the previous article "Artificial Neural Network (Artificial neural netwroks) Notes-Eliminate the sample order of the BP algorithm" to modify the weight of the method is called the "steepest descent method." Every time the weight of the changes are determined, the weight will be modified. Even to the simplest single layer perceptron.

But we have a question, whether every time the weight modification is good?

Although the "steepest descent method" can be mathematically proved to be more and more approximate to the optimal solution each time than the previous one, this approximation is an endless process. And in the face of the problem of local minima, the "steepest descent method" does not necessarily have enough energy to jump out of the local minimum, and may even sink deeper.

Therefore, some nondeterministic statistical methods are proposed.

The basic idea is: randomly select a join weight from the given network, a pseudo random adjustment quantity is proposed for the connection right, when the selected weights are modified with this adjustment, if the performance of the network is improved after being considered modified, the adjustment is retained otherwise.

The specific algorithm flow is as follows:

1 take a sample from the sample set S (x,y)

2 input x into the network, calculate the actual output O

3 Finding out the error measure of the network about Y,o E

4 randomly from W (1), W (2),.... In W (L), select a Join weight wij (P),

5 Generate a small random number wij (P)

6 modify Wij (p) with Wij (p),

7 with modified W (1), W (2),.... W (L) recalculate the actual output of the network O '

8 Find out the network about Y,o ' error measure e '

9 If E ' >e, then keep the W (1), W (2),.... W (L) modification, otherwise, according to the probability to determine whether this modification is useful, if useful, then keep the W (1), W (2),.... Modification of W (L), if this modification is considered useless, give it up

10 repeat the above process until the network meets the requirements.

How much is the modification of the connection weight better?

Too small: It is difficult to escape when falling to a local minimum.

Too large: Causes the jitter to return between the two local minimum points.

Solution: Control the size of the modification, the amount of modification from large to small (solve too big problem)

Allow the network to become temporarily bad (solve too small problem)

The energy of the network is mentioned above, which refers to the description in the simulated annealing algorithm.

The next article will describe how to use the simulated annealing algorithm to determine the fate of the modified amount.