Time series prediction using TensorFlow seq2seq

Time series prediction can be based on short-term forecasts, long-term forecasts and specific scenarios, such as Arma, ARIMA, neural network prediction, SVM prediction, grey prediction, fuzzy prediction, combined forecasting method and so on. The so-called no best model, only the most suitable model. As to which model can achieve the highest precision for a particular predictive problem, it needs to be proved by experiments. In this paper, a single Variable time series prediction experiment is carried out by using the TensorFlow SEQ2SEQ model to understand the SEQ2SEQ model infrastructure and the ability of validating the model in Non-linear data. The model with attention mechanism is not considered here for the time being, which can refer to the neural Machine translation by jointly Learning to Align and Translate.
The most basic SEQ2SEQ model consists of three parts, namely encoder, decoder, and the intermediate state vectors of the connection, encoder by learning input, encoding it into a fixed-size state vector s, and then transmitting s to decoder, The decoder is then output by learning the state vector s. The following figure is a model based on LSTM for both encoder and decoder.

First we import the packages we need:

Import TensorFlow as TF
import numpy as NP 
import random
import math from
matplotlib import Pyplot as plt< C4/>import os
Import copy

1. Data preparation
First, we generate a series of samples without noise:

x = Np.linspace (0,
gen) y = 2 * np.sin (x)

L1, = Plt.plot (x[:85], y[:85], ' y ', label = ' training samples ')
L2, = Plt.plot (x[85:], y[85:105], ' c--', label = ' test samples ')
plt.legend (handles = [L1, L2], loc = ' upper left ') 
  plt.show ()

To simulate real-world data, we add some random noises:

train_y = Y.copy ()

noise_factor = 0.5
train_y + + NP.RANDOM.RANDN (+) * Noise_factor

L1, = Plt.plot (x[:85), t Rain_y[:85], ' yo ', label = ' training samples ')
Plt.plot (x[:85], y[:85], ' y: ')
L2, = Plt.plot (x[85:], train_y[85: ], ' co ', label = ' test samples ')
Plt.plot (x[85:], y[85:], ' C: ')
plt.legend (handles = [L1, L2], loc = ' upper Left ')
Plt.show ()

Then we set the input output sequence length and generate the training sample and test sample:

 Input_seq_len = Output_seq_len = x = Np.linspace (0, a) train_data_x = x[:85] def true_signal (x): Y = 2 * Np.sin (x) return y-def noise_func (x, noise_factor = 1): Return Np.random.randn (len (x)) * Noise_factor def  Generate_y_values (x): Return true_signal (x) + Noise_func (x) def generate_train_samples (x = train_data_x, batch_size = Input_seq_len = Input_seq_len, Output_seq_len = Output_seq_len): total_start_points = Len (x)-Input_seq_len-o Utput_seq_len Start_x_idx = Np.random.choice (range (total_start_points), batch_size) input_seq_x = [X[i: (i+input_s Eq_len)] for I-start_x_idx] output_seq_x = [x[(i+input_seq_len):(I+input_seq_len+output_seq_len)] for I in start_x_ IDX] input_seq_y = [Generate_y_values (x) for x in input_seq_x] output_seq_y = [Generate_y_values (x) for x in OUTP Ut_seq_x] #batch_x = Np.array ([[True_signal ()]]) return Np.array (input_seq_y), Np.array (output_seq_y) 

Input_seq, Output_seq = Generate_train_samples (batch_size=10)

Visualize the noise-containing data:

results = [] for
I in range (m):
    temp = generate_y_values (x)
    results.append (temp)
results = Np.array ( Results) for

I in range (m):
    L1, = Plt.plot (Results[i].reshape (1), ' co ', LW = 0.1, alpha = 0.05, label = ' n Oisy training Data ')

L2 = Plt.plot (true_signal (x), ' m ', label = ' hidden True signal ')
plt.legend (handles = [L1, l 2], loc = ' lower left ')
plt.show ()

2. Establish a basic RNN model
2.1 Parameter Settings

# # Parameters
learning_rate = 0.01
Lambda_l2_reg = 0.003  

# network Parameters
# Length of input signals< C4/>input_seq_len = 
# Length of output signals
Output_seq_len = 
# size of lstm Cell
Hidden_dim = 64< c9/># num of input signals
Input_dim = 1
# num of output signals
Output_dim = 1
# num of stacked Lstm Layers 
Num_stacked_layers = 2 
# gradient clipping-to Avoid gradient exploding

2.2 Model Architecture
The SEQ2SEQ model here is basically consistent with the model provided by TensorFlow in GitHub.

def build_graph (feed_previous = False): tf.reset_default_graph () Global_step = tf.
                  Variable (initial_value=0, name= "Global_step", Trainable=false, COLLECTIONS=[TF. Graphkeys.global_step, TF.
                               Graphkeys.global_variables]) weights = {' Out ': tf.get_variable (' weights_out ', \
                               Shape = [Hidden_dim, Output_dim], \ dtype = Tf.float32, \
                               initializer = Tf.truncated_normal_initializer ()),} biases = {' Out ': tf.get_variable (' biases_out ', \
                               Shape = [Output_dim], \ dtype = Tf.float32, \  initializer = Tf.constant_initializer (0.)), with Tf.variable_scope (' Seq2seq '): # encoder:
   Inputs ENC_INP = [Tf.placeholder (Tf.float32, shape= (None, Input_dim), name= "inp_{}". Format (t))            For T in range (Input_seq_len)] # decoder:target Outputs target_seq = [
        Tf.placeholder (Tf.float32, shape= (None, Output_dim), name= "y". Format (t)) for T in range (Output_seq_len) 
        ] # Give a ' go ' token to the decoder. # If DEC_INP are fed into decoder as inputs, this is ' guided ' training; Otherwise only the # A, would be the Fed as decoder input which is then ' un-guided ' DEC_INP = [TF. 
            Zeros_like (Target_seq[0], Dtype=tf.float32, name= "Go")] + target_seq[:-1] with Tf.variable_scope (' Lstmcell '): Cells = [] for I in range (num_stacked_layers): With Tf.variable_scope (' rnn_{} '. Form at (i): Cells.append (Tf.contrib.rnn.LSTMCell (hidden_dim)) cell = Tf.contrib.rnn.MultiRNNCe
               ll (cells) def _rnn_decoder (decoder_inputs, initial_state, Cell,         Loop_function=none, Scope=none): "" "RNN decoder for the Sequence-to-sequence mo
          Del.
            Args:decoder_inputs:a List of 2D tensors [Batch_size x input_size].
            initial_state:2d Tensor with shape [Batch_size x cell.state_size]. Cell:rnn_cell.
            Rnncell defining the cell function and size.  Loop_function:if not None, this function is applied to the i-th output in order to generate the I+1-st Input, and decoder_inputs'll is ignored, except for the "" "
              This can is used for decoding, but also for training to emulate http://arxiv.org/abs/1506.03099.
                Signature-Loop_function (prev, i) = next * prev is a 2D Tensor of shape [Batch_size x output_size], * I is a-integer, the step number (when advanced control is needed), * Next is a 2D tenso R of Shape [Batch_size x INPUt_size]. Scope:variablescope for the created subgraph;
          Defaults to "Rnn_decoder". Returns:a tuple of the form (outputs, state), where:outputs:a list of same length as Deco
              Der_inputs of 2D tensors with shape [Batch_size x Output_size] containing generated.
                State:the State of the all cell at the final time-step.
                It is a 2D Tensor of shape [Batch_size x cell.state_size]. (Note this in some cases, like basic RNN cell or GRU cell, outputs and states can is the same.
          They are different for lstm cells though.) "" with Variable_scope.variable_scope (Scope or "Rnn_decoder"): state = Initial_state ou Tputs = [] prev = None for I, INP in Enumerate (decoder_inputs): If Loop_function is Not None and Prev are not none:with variable_scope.variable_scope ("Loop_function ", reuse=true): InP = loop_function (prev, i) if I > 0:variable_scope. 
              Get_variable_scope (). Reuse_variables () output, state = cell (INP, state) outputs.append (output) If Loop_function is isn't None:prev = output return outputs, State def _b
                              Asic_rnn_seq2seq (encoder_inputs, decoder_inputs, Cell, Feed_previous, Dtype=dtypes.float32, SC
          Ope=none): "" "Basic RNN sequence-to-sequence model. This model is runs a RNN to encode encoder_inputs into the state vector, then runs, decoder, initialized and T
          He is encoder state and on decoder_inputs.
          Encoder and decoder Use the same RNN cell type, but don ' t share parameters. Args:encoder_inputs:a List of 2D tensors [baTch_size x input_size].
            Decoder_inputs:a List of 2D tensors [Batch_size x input_size]. Feed_previous:boolean;  If True, only the ' the ' of Decoder_inputs would be used (the ' go ' symbol), all other inputs would be generated By the previous decoder output using _loop_function below.
            If False, decoder_inputs are used as given (the standard decoder case).
            Dtype:the Dtype of the initial state of the RNN cell (default:tf.float32). Scope:variablescope for the created subgraph;
          Default: "Basic_rnn_seq2seq". Returns:a tuple of the form (outputs, state), where:outputs:a list of same length as Deco
              Der_inputs of 2D tensors with shape [Batch_size x Output_size] containing the generated outputs.
                State:the state of each decoder cell in the final time-step.
      It is a 2D Tensor of shape [Batch_size x cell.state_size].    "" with Variable_scope.variable_scope (Scope or "Basic_rnn_seq2seq"): Enc_cell = Copy.deepcopy (c
                ELL) _, Enc_state = Rnn.static_rnn (Enc_cell, encoder_inputs, Dtype=dtype) if feed_previous: Return _rnn_decoder (decoder_inputs, enc_state, Cell, _loop_function) Else:return _r Nn_decoder (decoder_inputs, Enc_state, cell) def _loop_function (prev, _): "' Naive implementation of loo p function for _rnn_decoder. Transform prev from Dimension [Batch_size x Hidden_dim] to [Batch_size x Output_dim], which would be u SED as decoder input of next time step "' Return Tf.matmul (prev, weights[' out ')" + biases[' out ') dec_o Utputs, dec_memory = _basic_rnn_seq2seq (ENC_INP, DEC_INP, cell, feed_p revious = feed_previous) reshaped_outputs = [Tf.matmul (i, weights[' out "]) + biases[' out '] for I inDec_outputs] # Training loss and optimizer with Tf.variable_scope (' loss '): # L2 Loss 

        = 0 for _y, _y in Zip (reshaped_outputs, target_seq): Output_loss + = Tf.reduce_mean (Tf.pow (_y-_y, 2))
            # L2 regularization for weights and biases Reg_loss = 0 for Tf_var in Tf.trainable_variables (): If ' Biases_ ' in Tf_var.name or ' weights_ ' in Tf_var.name:reg_loss + = Tf.reduce_mean (tf.nn.l2_ Loss (Tf_var)) loss = Output_loss + Lambda_l2_reg * Reg_loss with Tf.variable_scope (' Optimizer '): Opt
                Imizer = Tf.contrib.layers.optimize_loss (Loss=loss, Learning_rate=learning_rate,

    Global_step=global_step, optimizer= ' Adam ', clip_gradients=gradient_clipping) Saver = tf.train.Saver return dict (ENC_INP = enc_inp, Target_seq = target_seq, Train_op = Optimizer, loss=loss, saver = saver, reshaped_outputs = reshaped_outputs,) 

2.3 Model Training
Sets the batch size to 16, and the iteration number is 100.

Total_iteractions =
Batch_size =
keep_rate = 0.5
train_losses = []
val_losses = []

x = NP.LINSP Ace (0, a)
train_data_x = x[:85]

Rnn_model = build_graph (feed_previous=false)

saver = Tf.train.Saver (

init = Tf.global_variables_initializer () with
TF. Session () as Sess:

    sess.run (init)

    to I in range (total_iteractions):
        batch_input, batch_output = Generate_train_samples (batch_size=batch_size)

        feed_dict = {rnn_model[' enc_inp '][t]: batch_input[:,t].reshape ( -1,input_dim) for T in range (Input_seq_len)}
        feed_dict.update ({rnn_model[' target_seq '][t]: batch_output[:,t]. Reshape ( -1,output_dim) for T in range (Output_seq_len)})
        _, loss_t = Sess.run ([rnn_model[' Train_op '], rnn_model[' Loss '], feed_dict)
        print (loss_t)

    temp_saver = rnn_model[' saver '] ()
    Save_path = Temp_saver.save (sess , Os.path.join ('./', ' univariate_ts_model0 ')

print ("Checkpoint saved at:", Save_path)

The loss of the visible output is trending down:

57.3053
32.0934
52.935
37.3215
37.121
36.0812 24.0727 27.5016 27.8621 29.2104
25.2804
26.4493
24.8914
26.5799
24.9871
22.0373 24.2079
23.1084
24.7991
24.0452
25.4788
23.0237
22.9508
25.1181 27.6942 20.5367 21.2899
28.4336
21.9943
21.8568
26.5138
22.5602 21.1466 22.0203 25.0466
21.0588
23.0585
22.5311
26.1076
21.897
20.694 21.459
21.1082
20.9176
21.8146
21.0299
22.3609
20.4057
21.0788 23.2162 24.8537 22.1202
20.7458
22.5411
21.4423
23.8266
23.3528 22.5934 20.2631 21.4907
19.6011
20.5983
23.4366
25.6377
23.4134
20.6773 18.802
25.341
22.935
21.8413
22.1599
23.4422
24.0427
22.8963 22.336 20.397 22.0162
21.2389
20.5336
21.0605
22.8863
22.3839
22.1059 22.0032
20.505
23.4216
26.9626
22.8875
21.9535
22.4898
20.1734 19.936 19.7218
20.2067
19.4964
24.0207
21.0039
22.38 20.1875 Checkpoint saved at: 22.7191  ./univariate_ts_model0

3. Forecast
We use the model in the test set to predict

Test_seq_input = True_signal (train_data_x[-15:])

Rnn_model = Build_graph (feed_previous=true)

init = Tf.global_variables_initializer () with
TF. Session () as Sess:

    sess.run (init)

    saver = rnn_model[' saver '] (). Restore (Sess, Os.path.join ('./', ' Univariate_ Ts_model0 '))

    feed_dict = {rnn_model[' enc_inp '][t]: Test_seq_input[t].reshape (1,1) for T in range (Input_seq_len)}
    feed_dict.update ({rnn_model[' target_seq '][t]: Np.zeros ([1, Output_dim]) for T in range (Output_seq_len)})
    Final_preds = Sess.run (rnn_model[' reshaped_outputs '], feed_dict)

    final_preds = Np.concatenate (final_preds, Axis = 1)

The resulting forecast results are as follows:

L1, = Plt.plot (range (in), True_signal (train_data_x[:85)), label = ' training Truth ')
L2, = Plt.plot (range (in), y[ [], ' yo ', label = ' Test Truth ')
l3, = Plt.plot (range (in), Final_preds.reshape ( -1), ' ro ', label = ' Test predict Ions ')
plt.legend (handles = [L1, L2, L3], loc = ' lower left ')
plt.show ()

References
https://weiminwang.blog/2017/09/29/multivariate-time-series-forecast-using-seq2seq-in-tensorflow/
Http://baijiahao.baidu.com/s?id=1571869641858839&wfr=spider&for=pc
https://blog.csdn.net/leadai/article/details/78809788
https://www.zhihu.com/question/21229371
https://blog.csdn.net/thriving_fcl/article/details/74853556