transcoding data using Python

Big data is the most annoying is the poor quality of data, in order to import data into the SEQUOIADB, need to require the text is UTF-8 mode, using eNCA to view the file encoding is gb2312, and then eNCA to utf-8 error. Google, the whole planet doesn't know why, try to transcode with Python

#-*-coding:utf-8-*-ImportCodecsImportSYSPrint "file name:", sys.argv[1]filename= Sys.argv[1]if(FileName = =None): Exit (1) file=open (filename) WriteFile= Open (filename+"UTF8","w+") BOM= File.read (3)if(BOM = =codecs. Bom_utf8): File.seek (3) forAinchFile:writefile.write (A.decode ("gb2312"). Encode ("Utf-8")) File.close () writefile.close ()

Here are two points: 1:for line in file is Python to read large files faster Method 2: The above code transcoding times wrong: unicodedecodeerror: ' gb2312 ' codec can ' t decode bytes in Positio N 2-3: Illegal multibyte sequence The second problem: Writefile.write into print does not have this problem, but a write there is a problem, very confusing, and then found an article on the internet said: the processing of characters is really gb2312, But the inclusion of some of the special characters in the gb2312 can not find the relevant mapping into Unicode, the use of GB18030, he is a superset of GB2312, contains a lot of special characters is gb2312 No, changed to GB18030 can be. However, there is still no reason why print can work.

transcoding data using Python