Python: Convert UTF-8 files into gbk files,

Source: Internet
Author: User

Python: Convert UTF-8 files into gbk files,
Requirement: the code for converting a UTF-8 file to a gbk file is as follows:

def ReadFile(filePath,encoding="utf-8"):    with codecs.open(filePath,"r",encoding) as f:        return f.read() def WriteFile(filePath,u,encoding="gbk"):    with codecs.open(filePath,"w",encoding) as f:        f.write(u) def UTF8_2_GBK(src,dst):    content = ReadFile(src,encoding="utf-8")    WriteFile(dst,content,encoding="gbk")
Code Description: The second parameter of the ReadFile function specifies to read files in UTF-8 format. The returned content is Unicode and then Unicode is written into files in gbk format. In this way, the requirements can be met. However, if the file to be converted contains some characters that are not included in the gbk character set, an error is returned, similar to the following: UnicodeEncodeError: 'gbk' codec can't encode character U' \ xa0' in position 4813: illegal multibyte sequenceThe preceding error message indicates that Unicode U' \ xa0' cannot be encoded into gbk when Unicode is encoded as gbk. Here, we need to figure out the relationship between gb2312, gbk and gb18030 GB2312: 6763 Chinese characters GBK: 21003 Chinese characters GB18030-2000: 27533 Chinese characters GB18030-2005: 70244 Chinese characters so, GBK is the superset of GB2312, and GB18030 is the superset of GBK. After clarifying the relationship, we can further improve the Code:
def UTF8_2_GBK(src,dst):    content = ReadFile(src,encoding="utf-8")    WriteFile(dst,content,encoding="gb18030")

After running, no error is reported and it can run properly.

Because the characters corresponding to U' \ xa0' can be found in the GB18030 character set. In addition, there is another Implementation Scheme: The WriteFile method needs to be modified.
def WriteFile(filePath,u,encoding="gbk"):    with codecs.open(filePath,"w") as f:        f.write(u.encode(encoding,errors="ignore"))

Here, we convert Unicode encoding (encode) into gbk format, but note that the second parameter of the encode function is assigned "ignore", indicating that during encoding, ignore characters that cannot be encoded,

The same is true for decoding. However, after execution, we found that UTF-8 files can be successfully modified to the ansi format. However, in the generated file, each row has a blank line. Here, you can specify the file to be written as a binary stream. The modified code is as follows:
def WriteFile(filePath,u,encoding="gbk"):    with codecs.open(filePath,"wb") as f:        f.write(u.encode(encoding,errors="ignore"))
Related Articles: http://www.crifan.com/python_csv_writer_writerow_redundant_new_line/
Http://www.v2ex.com/t/40033

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.