"Python" python splits Chinese and non-Chinese in text analysis

1. Description of the problem

For text analysis, the Chinese and non-Chinese are processed separately, and the Chinese part of the text is extracted by Python for the required processing.

2. Problem solving

Development environment: Linux

The program code is as follows: split.py

#!/usr/bin/python#-*-coding:utf-8-*-import sysreload (SYS) sys.setdefaultencoding ("UTF8") import re                                   #导入正则表达式模块: RE module def translate (Inputfile, outputFile): fin = open (inputfile, ' R ') #以读的方式打开输入文件fout = open (OutputFile, ' W ') #以写的方式打开输出文件for eachline I N fin: #按行读入文件内容line = Eachline.strip (). Decode (' utf-8 ', ' ignore ') #处理前进行相关的 Processing, including converting to Unicode, etc. p2 = re.compile (ur ' [^\u4e00-\u9fa5] ') #中文的编码范围是: \u4e00 to \u9fa5zh = "". Join (P2.split (Lin e). Strip () zh = ",". Join (Zh.split ()) outstr = zh #经过相关处理后得 To Chinese text fout.write (Outstr.strip (). Encode (' utf-8 ') + ' \ n ') fin.close () fout.close () if __name__ = = ' __main__ ': Translate ( SYS.ARGV[1], sys.argv[2]) # #通过获得命令行参数获得输入输出文件名来执行, convenient

After the program is complete, enter it on the Linux command line: Python split.py myinput.txt myoutput.txt

will be able to execute. The final translation results are written to the output file Myoutput.txt.

3. Attention Issues

(1) The 16th line, the Chinese encoding range is: \u4e00 to \u9fa5, so the line of [^\U4E00-\U9FA5] is non-Chinese, that is, according to non-Chinese cut out Chinese text.

I hope we have some help, thank you.

"Python" python splits Chinese and non-Chinese in text analysis