008---recurrent neural network-rnn

Conventional neural networks:

Conventional neural networks are each counted, and each x is independent of each other.

Recurrent neural Networks:

A recursive neural network to do one thing is, in the hidden layer, x1 the use of the x0 w,x2 using the w,x3 of the X2 to use the W ... And so on

For example, the following example:

X0 says "I was born."

X1 says "in"

X2 says "China"

XT says, "I said."

Xt+1 would have predicted "Chinese."

RNN Network Details:

Both U and W are parametric matrices, and V is the fully connected matrix.

What RNN does with CNN is that each CNN input and output is independent, rnn each input depends on the output of the previous input.

So the RNN network next need the previous step of the useful parameters, rnn do natural language processing is appropriate,

Because the human language context is very important to combine.

RNN Network Reverse propagation:

Because of the link characteristics of the RNN network, the weight value update for each input parameter updates the previous input parameter, and CNN is only updating its own input parameters.

Limitations of the RNN network

1, the network is too large

For example:

I am Chinese, ..... I Love China

If there are 500 words in the ellipsis, then the original AM is not a big part of the last China,

But RNN will use the first am also for the last China

2, it is possible for gradients to disappear:

If the network is large, when the parameter is updated, the chain law gradient forward, if the gradient is very small, then the gradient after the front gradients will be smaller and less, approaching 0

008---Recursive neural network-rnn