Python Coding issues

The representation of a string inside Python is Unicode encoding, so in encoding conversion, it is usually necessary to use Unicode as the intermediate encoding, that is, decoding the other encoded string (decode) into Unicode first. From Unicode encoding (encode) to another encoding.

The role of Decode is to convert other encoded strings into Unicode encodings, such as Str1.decode (' gb2312 '), to convert gb2312 encoded string str1 into Unicode encoding.

The role of encode is to convert Unicode encoding into other encoded strings, such as Str2.encode (' gb2312 '), to convert Unicode encoded string str2 to gb2312 encoding.

Therefore, the transcoding must first understand, the string str is what encoding, and then decode into Unicode, and then encode into other encodings.

The default encoding of the string in the code is consistent with the encoding of the code file itself.

such as: s= ' Chinese '

If it is in a UTF8 file, the string is UTF8 encoded, and if it is in a gb2312 file, it is encoded as gb2312. In this case, to encode the conversion, you need to first convert it to Unicode encoding using the Decode method, and then use the Encode method to convert it to another encoding. Typically, you create a code file by using the system default encoding when you do not specify a specific encoding method.

If the string is defined like this: S=u ' Chinese '

The encoding of the string is specified as Unicode, which is the internal encoding of Python, regardless of the encoding of the code file itself. Therefore, for this case to do the encoding conversion, only need to directly use the Encode method to convert it to the specified encoding.

If a string is already Unicode, then decoding will be an error, so it is common to determine whether it is encoded as Unicode:

Isinstance (S, Unicode) #用来判断是否为unicode

Encode with a non-Unicode encoded form of STR will error

How do I get the default encoding for my system?

#!/usr/bin/env python

#coding =utf-8

Import Sys

Print sys.getdefaultencoding ()

The program is printed on the English windowsxp as: ASCII

Set the default encoding of strings within a document

Import Sys

Reload (SYS)

Sys.setdefaultencoding (' Utf-8 ')

Set Python global default encoding simply create a new sitecustomize.py under the Python installation directory/lib/site-packages folder

Add the following code:

# Encoding=utf8

Import Sys

Reload (SYS)

Sys.setdefaultencoding (' UTF8 ')

Note that the sitecustomize.py name cannot be modified.

Using Unicode (str, ' gb2312 ') with the Str.decode (' gb2312 ') effect is the same as decoding gb2312 encoded STR into Unicode encoding

Use str.__class__ to view the encoded form of STR

>>> Import Sys

>>> sys.getdefaultencoding ()

' ASCII '

>>> s = "Chinese"

>>> s.__class__

<type ' str ' >

>>> isinstance (s,unicode) # is Unicode code

False

>>> s

' \xd6\xd0\xce\xc4 '

>>> Print S

Chinese

>>> S.decode ("gb2312") # decoded to Unicode

U ' \u4e2d\u6587 '

>>>

>>> s2 = u "Chinese"

>>> s2.__class__

<type ' Unicode ' >

>>> isinstance (S2, Unicode)

True

>>> S2

U ' \u4e2d\u6587 '

>>> Print s2

Chinese

>>> S2.encode ("gb2312") # Encoded as gb2312

' \xd6\xd0\xce\xc4 '

>>>

The following example is the output in gb2312 format, running in idle is not effective

#!/usr/bin/env python

#coding =utf-8

Import Sys

Reload (SYS)

Sys.setdefaultencoding (' UTF8 ')

s= "Chinese"

If Isinstance (S, Unicode):

Print S.encode (' gb2312 ')

Else

Print S.decode (' utf-8 '). Encode (' gb2312 ')

Raw_input ("press ENTER")

In addition, there is a conversion tool in the Django library that is very useful to convert any encoding

From django.utils.encoding import Smart_str, Smart_unicode

s = "Chinese"

S= Smart_str (S, encoding= "Utf-8") # Default conversion to UTF-8 encoding

If you want to convert Unicode, you can use

S= Smart_unicode (S, encoding= "Utf-8") # Default conversion to UTF-8 encoding

Python also provides a codecs encoding conversion library

>>> Import Codecs

>>> f = codecs.open ("/data/testdata/test.txt", "R", "Utf-8") # Open file in utf-8 form

>>> for line in F:

If line[:3] = = codecs. Bom_utf8:

line = line[3:]

Print Line

Python Coding issues