Use the KENLM Training statistical language model and use the model (based on Linux14.04) __linux

Reference Documentation:

http://thegrandjanitor.com/2015/12/28/using-arpa-lm-with-python/

Https://www.52ml.net/4816.html

Http://www.cnblogs.com/apexchu/p/5015961.html

http://blog.csdn.net/zh515858237/article/details/54692486

http://blog.csdn.net/fxjtoday/article/details/6059874

I have seen the neural network language model before, but I wrote the code is not ideal. The language model is used in the group, so the KENLM is chosen after the research and comparison. The reason is: KENLM is a C + + written language model, KENLM better than srilm, to support a single large data training. BERKELEYLM is written in Java, claiming to be similar to KENLM, but I am not familiar with Java.

The installation error raises a series of other problems, and of course it learns something.

Problem Discovery and Learning (1) Mixed programming---python and C + +:

We need to install the Cython compiler on the machine. The PIP install Cythton can be installed. Or use https://github.com/cython/cython/wiki/Installing for source code installation. Cython code is not the same as Python code and must be compiled. The compilation process is divided into two phases: <1>cython compilation. Pyx (. pxd) is a. C or. cpp file. The <2>c compiler compiles the. c/.cpp file into a. So file (Linux version) or a. pyd file (Windows version). A specific step can refer to http://blog.csdn.net/daniel_ustc/article/details/77622895 or http://www.jianshu.com/p/0dffe9aeba29. or http://www.jyguagua.com/?p=2868 to do it. Step Two we can refer to http://blog.csdn.net/acb0y/article/details/6553051 or http://blog.csdn.net/qq_33850438/article/details/52014399. After step two is complete, we can make a call to the generated. So or. pyd file in the Python code.

Problem Discovery and Learning (2)--Install KENLM and use:

At first I was using the official website guide http://kheafield.com/code/kenlm/to install (after all, the official). and refer to the http://blog.csdn.net/zh515858237/article/details/54692486, http://www.osyunwei.com/archives/5530.html, and < Installation bzlib: Open the URL http://www.bzip.org→ download the latest version of the bzlib→ extract to the local → Bzlib in the decompression path to execute Make,sudo make install > for the earlier dependent environment configuration. But after installation, the example.py of the running sample shows: Importerror:no module named ' KENLM '. At that time I foolishly thought that the kenlm.cpp of the same directory needs to be converted into kenlm.so to be invoked. But until I saw http://thegrandjanitor.com/2015/12/28/using-arpa-lm-with-python/, I realized: The original KENLM did not install successfully ah ... Follow this blog and install the environment that you might have installed before. After that, on the Linux command line import KENLM, this time normal. Then, refer to Https://www.52ml.net/4816.html, https://www.52ml.net/4813.html, http://www.cnblogs.com/zidiancao/p/6067147. HTML, http://www.pythontip.com/blog/post/7217/, https://www.52ml.net/4813.html model training and model use. For the model evaluation of the language model, the following will continue to be updated in this article.