LSTM Network (Long short-term Memory)

This paper is based on the first two, multilayer perceptron and its BP algorithm (multi-layer Perceptron) and recurrent neural network (recurrent neural networks,rnn)

RNN has a fatal flaw, the traditional MLP also has this flaw, before looking at this flaw, the first to sacrifice the RNN's reverse conduction formula with the MLP's reverse conduction formula:

\[RNN: \ \delta_h^t = f ' (a_h^t) \left (\SUM_K\DELTA_K^TW_{HK} + \sum_{h '} \delta^{t+1}_{h '}w_{hh '} \right) \]

\[MLP: \ \delta_h = f ' (a_h) \sum_{h ' =1}^{h_{l+1}} w_{hh '}\delta_{h '}\]

Note that when the residuals are passed back in the time dimension, each passing moment leads to a large attenuation of the signal, and why, because of this non-linear activation function $f $, the general shape of this function is as follows:

As shown, the activation function $f $ in the gradient outside the red line is very small, so the function $f the derivative $f ' $ is small, and after the residuals are listed above the inverse pass formula can be derived, each passing a moment, attenuation $f ' $ order of magnitude, so after multiple moments will cause the time dimension gradient exponential attenuation, The feedback signal at the moment does not affect the distant past. Therefore, although RNN can theoretically handle sequences of any length, in an internship application, RNN is even difficult to handle sequences of more than 10 lengths. The image depicts the phenomenon of this gradient disappearing.

It can be seen that for $t =1$, the feedback from $t >1$ time is getting smaller, and the depth of the color in the figure represents the size of the feedback signal. In order to solve the phenomenon of gradient disappearance, scholars have studied many methods, of which Long short-term Memory performance is the most outstanding.

LSTM Network (Long short-term Memory)