' Illegal multibyte Sequen ' error during Python transcoding

Source: Internet
Author: User
Tags error handling function prototype

In Python, you can invoke the decode and encode methods on a string to implement transcoding.

For example, to convert a string object s from GBK inside code to UTF-8, you can do the following

S.decode (' GBK '). Encode (' utf-8′)
However, in real-world development, I found that this approach often occurs abnormally:
Unicodedecodeerror: ' GBK ' codec can ' t decode bytes in position 30664-30665:illegal multibyte sequence
This is because illegal characters are encountered-especially in some programs written in C + +, full-width spaces are often implemented in many different ways, such as \xa3\xa0, or \xa4\x57, which appear to be full-width spaces, but they are not "legal" full-width spaces (true em spaces are \XA1\XA1), so an exception occurred during transcoding.
This is a headache because as long as there is an illegal character in the string, the entire string-sometimes the entire article-cannot be transcoded.

Workaround:
S.decode (' GBK ', ' ignore '). Encode (' utf-8′ ')
Because Decode's function prototype is decode ([encoding], [errors= ' strict ']), a second parameter can be used to control the policy of error handling, the default parameter is strict, which represents an exception thrown when an illegal character is encountered;
If set to ignore, illegal characters are ignored;
If set to replace, it will replace illegal characters;
If set to Xmlcharrefreplace, the character reference of the XML is used.

Python documentation

Decode ([encoding[, errors]])
Decodes the string using the codec registered for encoding. Encoding defaults to the default string encoding. Errors May is given to set a different error handling scheme. The default is ' strict ', meaning that encoding errors raise unicodeerror. Other possible values is ' ignore ', ' replace ' and any other name registered via Codecs.register_error, see section 4.8.1.


' Illegal multibyte Sequen ' error during Python transcoding

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.