nlp-use rnn/lstm to do text generation _lstm

Note: Learn notes, content from July online video-author plus one, with memory neural network

Text generation, no direct feed feeds, we want our classifiers to remember contextual relationships, and RNN's purpose is to allow information that has sequential relationships to be considered. Sequential relationship is the relationship of information in time.

1.RNN

2.lstm-Enhanced Edition RNN
Description

3. Analog information changes in lstm ① forget the door

Determines what information we should forget.
② Memory Door

which should remember
③ Update Door

Update the old cell state to the new cell state
④ loses out.

By the memory to decide what door to output
Second, use lstm to do text generation

1. Expected note: A variety of Chinese language corpus can be found on the Internet, English novel Corpus can be downloaded from Project Gutenberg website txt text
2. Text generation: Generated at the char level or at word level
3. Code step: First step, import the required libraries

Import OS
import NumPy as NP
import NLTK from
keras.models import sequential from
keras.layers import Den SE from
keras.layers import dropout to
keras.layers import lstm from
keras.callbacks Import Modelcheckpoint from
keras.utils import np_utils from
Gensim.models.word2vec import Word2vec

Second step, text read into

Raw_text = ' for
file in Os.listdir (' ... /input/"):
    if File.endswith (". txt "):
        Raw_text + = open (". /input/"+file", errors= ' ignore '). Read () + ' \ n \ na '
# raw_text = Open (' ... /input/winston_churchil.txt '). Read ()
raw_text = Raw_text.lower ()
sentensor = Nltk.data.load (' tokenizers/ Punkt/english.pickle ')        
sents = sentensor.tokenize (raw_text)
corpus = [] For
Sen in Sents:
    Corpus.append (Nltk.word_tokenize (SEN))

The third step, w2v stew

W2v_model = Word2vec (Corpus, size=128, window=5, min_count=5, workers=4)

The fourth step is to process our training data and turn the source into a long x so that Lstm learns to predict the next word

Raw_input = [item for sublist in Corpus for item in sublist]
text_stream = []
vocab = W2v_model.vocab for
w Ord in Raw_input:
    if Word in vocab:
        text_stream.append (word)
len (text_stream)

The fifth step, constructs the training test set

Seq_length =
x = []
y = []
for I in range (0, Len (text_stream)-seq_length):

    given = Text_stream[i:i + Seq_length]
    predict = text_stream[i + seq_length]
    x.append (Np.array ([W2v_model[word] for word in given]
    ) Y.append (W2v_model[predict])
# We can look at the good data set's looks
print (x[10])
print (Y[10)

Sixth step, ① we already have a digital expression for input (W2V), we want to change it into lstm need array format: [Sample number, time step, feature],,② for output, we directly use 128-D output

x = Np.reshape (x, ( -1, Seq_length, 128))
y = Np.reshape (Y, (-1,128))

The seventh step, lstm model construction

Model = sequential ()
Model.add (LSTM (256, dropout_w=0.2, dropout_u=0.2, input_shape= (seq_length, 128)
) Model.add (Dropout (0.2)) Model.add (dense (128, activation= '
sigmoid '))
model.compile (loss= ' MSE ', Optimizer= ' Adam ')

Step eighth, run the model

Model.fit (x, Y, nb_epoch=50, batch_size=4096)

Finally, the model effect of the program test training

 #①def Predict_next (input_array): x = Np.reshape (Input_array, ( -1,seq_length,128)) y = model.predict (x) Return y def string_to_index (raw_input): Raw_input = Raw_input.lower () Input_stream = Nltk.word_tokenize (raw_inpu T) res = [] for word in input_stream[(len (input_stream)-seq_length):]: Res.append (w2v_model[word)) ret Urn Res def y_to_word (y): Word = W2v_model.most_similar (positive=y, topn=1) return word #②def generate_article ( Init, rounds=30): in_string = Init.lower () for I in range (rounds): n = Y_to_word (Predict_next (string_to_in Dex (in_string)) in_string + + n[0][0] return in_string #③init = ' Language Models allow us to measure H ow likely a sentence is, which are an important for Machine ' article = generate_article (init) print (article)