Code address for this section
Https://github.com/vic-w/torch-practice/tree/master/rnn-timer
RNN full name Recurrent neural network (convolutional neural Networks), which is a memory function by adding loops to the network. The natural language processing, image recognition and other aspects have far-reaching effects.
This time we're going to use RNN to implement a neural network timer, given a length of time, it waits until the end of time and then switches its state.
It is very simple to implement a timer in C language. We're going to write about this:
void timer (int delay_time)
{for
(int i=0; i<delay_time; i++)
{
delay (1);
}
return;
}
But how do you do that with neural networks? We can think of the RNN network as a black box with an input signal and an output signal. I want to make the input and output conform to this relationship:
Blue is the input, and green is the output. When the input signal produces a pulse, the timer starts to work, and the length of the timing is determined by the height of the pulse. When the timer is working, the output is 1, and when it stops working, the output is 0.
To implement this function, the module must have a storage unit inside it to record its status. At the very least, a mechanism equivalent to a temporary variable is required to record the length of time that has elapsed. This is the mission that RNN needs to accomplish.
If we want to train rnn neural network, we must first have enough data. Fortunately, our data can be generated indefinitely. I use Data_gen.lua to generate large amounts of data, each with a length of 100 and stored in the Data.t7 file in the same directory.
Data_gen.lua need to be executed separately
Th Data_gen.lua
With a lot of data, it's time to design our recursive neural network.
This time the network is also very simple, there is an input node on the left, the middle is a hidden layer with 20 nodes, the right is an output node. Unlike multilayer perceptron, the 20 nodes in the hidden layer in the middle of the RNN are connected to all nodes of the layer (including itself). So the number of connections that feed back to themselves has a total of 20*20=400 bars.
In order to implement such a network in torch, we need to use the RNN library. If you have not previously installed it, you can enter it at the command line:
Luarocks Install RNN
In the program, the first is still to contain the necessary library files.
Require ' RNN '
require ' gnuplot '
Next define some constants
BatchSize = 8--mini Batch size
rho =--rnn The maximum length of time to consider when training, which is also the number of times the back propagation through is going through
hiddensize = 20--The number of nodes in the middle hidden layer
Next, define the structure of the hidden layer:
R = nn. Recurrent (
hiddensize, nn. Linear (1, hiddensize),
nn. Linear (Hiddensize, hiddensize), nn. Sigmoid (),
Rho
)
The type of hidden layer R is nn. Recurrent. The following parameters are followed in turn:
1. The number of nodes contained in this layer is hiddensize
2. The previous layer (that is, the input layer) to the connection to this layer. Here is a linear connection with an input of 1 and an output of hiddensize.
3. The feedback connection of this layer node to itself. Here is an input of hiddensize, and the output is also a hiddensize linear connection.
4. The activation function used by this layer for input and feedback connections. Sigmoid is used here.
5. The maximum number of times the back propagation through time. Here is Rho = 100
Next, define the structure of the entire network:
RNN = nn. Sequential ()
rnn:add (nn. Sequencer (R))
Rnn:add (nn. Sequencer (NN. Linear (hiddensize, 1)))
Rnn:add (nn. Sequencer (NN. Sigmoid ()))
First define a container and then add the hidden layer r that you just defined. Then add the hidden layer to the output layer of the connection, where the input is 20, the output is a 1 linear connection. Finally, a layer of sigmoid function is connected.
Here in the definition of the network, each specific module is the use of NN. The parentheses of the sequencer are enclosed. Nn. Sequencer is a modified module. All through NN. Sequencer packaged modules become acceptable for sequence input.
For example, suppose a module could have accepted a 2-D tensor as input and output another 2-dimensional tensor. If we want to input a series of 2-D tensor into this module sequentially, we need to write a for loop. With the NN. Sequencer's decorations don't have to be so troublesome. Just put this series of 2-dimensional tensor unified into a large table, and then disposable to the NN. Sequencer on the line. Nn. Sequencer will put the tensor in the table into the network, and the network output of the tensor in turn into a large table back to you.
Define the network, and then define the criteria for the evaluation:
Criterion = nn. Sequencercriterion (NN. Msecriterion ())
The next thing is a routine. Propagate forward, propagate backward, update parameters ...
Batchloader = Require ' minibatchloader '
loader = batchloader.create (batchsize)
lr = 0.01
i = 1 for
n= 1,6000 do
--prepare inputs and targets
local inputs, targets = Loader:next_batch ()
local outputs = Rnn:forwar D (inputs)
Local err = Criterion:forward (outputs, targets)
print (i, err/rho)
i = i + 1
local gradoutputs = Criterion:backward (outputs, targets)
Rnn:backward (inputs, gradoutputs)
rnn:updateparameters (LR)
Rnn:zerogradparameters ()
End
It is important to note the format of the input and output data. I used Minibatchloader (the Minibatchloader.lua file in the same directory) to read the data from the DATA.T7, each time I read 8 sequences, the length of each sequence was 100. So the type of inputs in the Code is table, which has 100 elements, each of which is a 2-D, 8-column, 1-row tensor. During training, the first data of each of the 8 sequences in mini batch goes into the network together, followed by 8 data entered in the second row, so iterative.
When the training is complete, use one of the group inputs to put it into the network to observe its output:
Inputs, targets = Loader:next_batch ()
outputs = Rnn:forward (inputs)
x={}
y={} for
i=1,100
do Table.insert (x,inputs[i][{1,1}])
Table.insert (y,outputs[i][{1,1}])
end
x = Torch. Tensor (x)
y = torch. Tensor (y)
gnuplot.pngfigure (' timer.png ')
gnuplot.plot ({x},{y})
Gnuplot.plotflush ()
The results are shown below:
Although there are still some deviations in the timing, the RNN timer can be counted as the basic learning of timing.
