Language detection and discrimination of text using LangID Toolkit in Python python text processing

1. Description of the problem

When using Python for text processing, sometimes the text contained in Chinese, English, Japanese, and other languages, sometimes can not be processed at the same time, it is necessary to determine the current text belongs to which language. There is a LangID toolkit in Python that provides this functionality, andLangID currently supports detection in 97 languages, which is very useful.

2. Code of the program

The following Python is a program code that invokes the LangID Toolkit for language detection and discrimination of text:

import LangID #引入langid模块def Translate (inputfile, outputFile): fin = open (                                  Inputfile, ' r ') #以读的方式打开输入文件fout = open (OutputFile, ' W ') #以写的方式打开输出文件for eachline in fin: #依次读入每一行line = Eachline.strip (). Decode (' Utf-8 ', ' ignore ') #去除每行的首位空格等 and unify into unicodelinetuple = Langid.classify (line) #调用langid来对该行进行语言检测if line                                         Tuple[0] = = "zh": #如果该行语言大部分为中文, no processing is done CONTINUEOUTSTR = line #如果该行语言为非中文, prepare the output fout.write (Outstr.strip (). Encode (' utf-8 ') + ' \ n ') #输出非中文的行, convert from Unicode to utf-8 output fin.close () Fout . Close () if __name__ = = ' __main__ ': #相当于main函数translate ("MyInputFile.txt", "myou TputFile.txt ") 

The above code is used to process a text that will be output to a new file in a row that does not belong to Chinese.

3. Note

9th, 10 lines of code, the output of langid.classify (line) is a two-tuple, the first of the two tuples represents the language of the text, such as: En for Chinese, en for English, and so on; the second item in the binary represents the proportion of the language in the text that belongs to the first.

We hope to help you.

Language detection and discrimination of text using LangID Toolkit in Python python text processing