Google Translate integrates neural networks: machine translation for disruptive breakthroughs

Selected from Google Analytics

Author: Quoc v. Le, Mike Schuster

The heart of the machine compiles

Participation: Wu Yu

Yesterday, Google published a paper on arxiv.org "Google's neural machine translation system:bridging the Gap between Human and machine translation" Introducing Google's neural machine translation System (GNMT), the heart of the day machine was translated and recommended to the website (www.jiqizhixin.com). Today, Google research Blog published articles on the study was introduced, but also announced the GNMT into a very difficult Chinese-English language translation production, has aroused great concern in the industry.

Ten years ago, we released Google Translate (Google Translate), the core algorithm behind this service is the phrase-based MT (pbmt:phrase-based machine translation). Since then, the rapid development of machine intelligence has brought a huge boost to our speech recognition and image recognition capabilities, but improving machine translation is still a challenging goal.

Today, we announce the release of the Google Neural MT (Gnmt:google Neural machine translation) system, which uses current state-of-the-art training techniques to achieve the greatest increase in machine translation quality so far. For details of all our findings, please refer to our paper "Google's neural machine translation system:bridging the Gap between Human and machine translation" (see end) [1].

A few years ago, we started using recurrent neural networks (rnn:recurrent neural Networks) to directly learn a mapping of an input sequence (such as a sentence in a language) to an output sequence (the same sentence in another language) [2]. The phrase-based machine learning (PBMT) breaks down input sentences into words and phrases, and then largely independently translates them, while a neural machine translation (NMT) regards the entire input sentence as the basic unit of translation. The advantage of this approach is that this approach requires less engineering than the previous phrase-based translation system. When it was first proposed, NMT achieved the accuracy comparable to phrase-based translation systems on a medium-sized public benchmark dataset.

Since then, researchers have proposed a number of techniques for improving NMT, including simulating external alignment models (external alignment model) to deal with rare words [3], using attention (attention) to align input words and output words [4]. and breaking words into smaller units in response to rare words [5,6]. Despite these advances, NMT's speed and accuracy have yet to reach the requirements of a production system like Google Translate. Our new paper [1] Describes how we overcame many of the challenges of getting NMT to work on very large datasets, and how we have built a system that is fast and accurate enough to deliver better translations for Google's users and services.

The following visualization shows GNMT's process of translating a Chinese sentence into an English sentence. First, the network encodes the words of the Chinese sentence into a list of vectors, each of which characterizes the meaning of all the words that have been read so far ("encoder (Encoder)"). Once the entire sentence is read, the decoder begins to work-one word ("Decoder (Decoder)") that generates an English sentence at a time. In order to generate the correct words at each step, the decoder focuses on the weighted distribution of Chinese vectors ("note (Attention)"), which is most relevant for generating English words, and the transparency of the blue links indicates the degree of attention of the decoder to a coded word.

Using the human-side comparison as a standard, the translation of the GNMT system achieves a significant improvement over the previous phrase-based production system. With the help of a bilingual human evaluator, we measured on sample sentences from Wikipedia and news sites: GNMT reduced translation errors by more than 55% to 85% in translations of several major language pairs.

In addition to publishing this research paper today, we also announced that we have invested GNMT in the production of a very difficult language pair (Chinese-English) translation. Now, the mobile version and the web version of Google Translate's Chinese-English translation has been used in 100% GNMT machine translation-about 18 million translations per day. GNMT's production deployment is the use of our Open Machine Learning Toolkit TensorFlow and our tensor processing units (tpu:tensor processing Units), which provide sufficient computational power to deploy these powerful GNMT models, while also satisfying Stringent latency requirements for Google Translate products. Chinese to English translation is one of more than 10000 language pairs supported by Google Translate, and we will continue to extend our gnmt to far more language pairs in the coming months.

Machine translation is far from being fully resolved. GNMT will still make major mistakes that human translators will never make, such as missing words and wrong translations of proper nouns or rare terms, and the context in which sentences are translated individually without regard to their paragraphs or pages. In order to bring better service to our users, we have more work to do. However, GNMT represents a major milestone. We want to celebrate it with many researchers and engineers who have contributed to this research in the past few years-whether it's from Google or the wider community.

Both the Google Brain team and the Google Translate team are involved in the project. Nikhil Thorat and Big picture also helped visualize the project.

paper: Google's neural machine translation system:bridging, the Gap between Human and machine translation

Abstract: Neural MT (Nmt:neural machine translation) is an end-to-end learning method for automated translation, which is expected to overcome the shortcomings of traditional phrase-based translation systems. Unfortunately, it is well known that the computational cost of training and translation reasoning for NMT systems is very high. In addition, most NMT systems have difficulty coping with rare words. These problems hinder the application of NMT in real-world deployments and services, because accuracy and speed are critical in real-world applications. In this outcome, we have proposed gnmt--Google's neural machine translation (Google's neural-translation) system to try to solve many of these problems. Our model consists of a deep LSTM network with 8 encoders and 8 decoders, which uses a note (attention) and a residual connection (residual connections). To improve parallelism and reduce training time, our attention mechanism connects the bottom of the decoder to the top layer of the encoder. In order to speed up the final translation, we used the low-precision operation in the inference calculation process. To improve the handling of rare words, we divide the word into a finite set of common sub-words (Sub-word) (the component of the word), which is both the input and the output. This approach provides the flexibility of the character (character) "-delimited models and the balance between the validity of the word (word)"-delimited models, the ability to handle the translation of rare words naturally, and the eventual improvement of the overall accuracy of the system. Our beam search technology (beam search technique) uses a length normalization (length-normalization) process and uses a coverage penalty (coverage penalty), It can motivate the generation of output sentences that are likely to cover all the words in the source sentence. On the WMT ' 14 English-French and English-German benchmarks, GNMT achieves results comparable to current best results. By comparing the human comparison in a single set of simple sentences, it reduces the average translation error by 60% compared to the phrase-based system that Google has put into production.

Reference Documents:

[1] Google's neural machine translation system:bridging the Gap between Human and machine translation, Yonghui Wu, Mike s Chuster, Zhifeng Chen, Quoc v. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, J Eff Klingner, Apurva Shah, Melvin Johnson, xiaobing Liu,łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto K Azawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol V Inyals, Greg Corrado, Macduff Hughes, Jeffrey Dean. Technical report, 2016.

[2] Sequence to Sequence learning with neural Networks, Ilya sutskever, Oriol vinyals, Quoc v. Le. Advances in neural information processing Systems, 2014.

[3] Addressing the rare word problem in neural machine translation, Minh-thang Luong, Ilya Sutskever, Quoc v. Le, Oriol Vi Nyals, and Wojciech Zaremba. Proceedings of the 53th Annual meeting of the Association for Computational Linguistics, 2015.

[4] Neural machine translation by jointly learning to Align and Translate, Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio. International Conference on learning representations, 2015.

[5] Japanese and Korean voice search, Mike Schuster, and Kaisuke Nakajima. IEEE International Conference on Acoustics, Speech and Signal processing, 2012.

[6] Neural machine translation of Rare Words with Subword Units, Rico Sennrich, Barry Haddow, Alexandra Birch. Proceedings of the 54th Annual meeting of the Association for Computational Linguistics, 2016.

Source:

Http://mp.weixin.qq.com/s?__biz=MzA3MzI4MjgzMw==&mid=2650719470&idx=1&sn= 3368dea980517ea7d7942967da2740dc&chksm= 871b0090b06c89863620be4e75c757940d03d8a43cd3c1d9a8309b6594c1bccd769cab193177&scene=21#wechat_redirect