Paper notes-neural machine translation by jointly learning to Align and Translate

A attention mechanism is proposed for machine translation.

Background: RNN-based machine translation

The basic idea is to encode the language X first encoder, then decode decoder for language Y. Encoder and decoder can be seen as two layers of rnn? Coded hidden layer h and decoding the hidden layer s

RNN Encoder-decoder:

1) The input sentence, expressed as , through the loop layer to get hidden layer , the vector c is represented as a hidden layer of the function, C is the input encode out of the vector.

2) Next is the decoder stage, predicting the next word based on the previously predicted translated words and the input encoder

The innovation of this paper:

The conditional probabilities of the (2) type are rewritten, and the encoder of each yi,context are different and recorded as CI

About CI calculation: CI is represented as a series of hi linear weights, where hi is the hidden layer of the encoder end, defined as Annotation,hi (personal understanding) as the input of the word "I" near the information (the expression of the input terminal I is simply)

Alpha factor:

Alpha or E represents the annotation of the J-input word and the importance of the i-1 hidden state of the decoder end, so that the resulting CI will pay attention for some locations, equivalently as the translation word I to the original input some position pay Attetnion

Using BIRNN:

This paper uses bidirectional rnn to catch the forward and backward hi stitching together, so that the annotation can represent the information around the input word I.

Network structure:

Paper notes-neural machine translation by jointly learning to Align and Translate