Python character encoding (python2.6)

Very lazy, written a long time ago. has not been posted up.

Send it today, and later on in detail several encodings in memory condition

Problem:

Multi-system data interaction between different platforms, the system encoding format is different, general conditions such as Windows GB2312,SVN Utf-8, as well as possible MySQL GBK default encoding. The interaction of data between the three people requires character transcoding. Invoking the decode and encode decoding and encoding of the string module in Python, the following describes the Python character conversion and the actual problems we encounter in the project

Python encoding decoding:

1. Two types of Python encoding: STR and Unicode

There are two types of Python strings, one is STR and one is a Unicode type, where there are many encoding formats in the STR string. Type ("str"). __name__ can output string types, but there is no way to output str encoding format, Python's third-party library chardet enough of the relevant functions, you can judge the encoding format of Str.

Example: Import Chardet
Chardet.detect (RawData)
{' confidence ': 0.98999999999999999, ' encoding ': ' GB2312 '}

1) Str

The coding format of STR is gbk,utf8,latin1 and so on, we need to determine what kind of encoding str is before decode.

2) Unicode

The representation of a string inside Python is Unicode encoding, which usually requires Unicode as the intermediate encoding, i.e. decoding the other encoded string (decode) to Unicode, and then from Unicode encoding (encode) to another encoded string. Python reads and writes text, both read and write STR strings, and if a Unicode string is written, Python internally converts Unicode into a system encoding and then writes it to a file instead of writing Unicode or reading Unicode. No, Unicode is just an intermediate encoding, not a memory encoding.

1. The character conversion function uses

Decoding:

Decode: Converting other encoded strings to Unicode encoding

Eg:String.Decode ("GBK"), decodes the string "GBK" into a Unicode string

(misunderstanding: The string that converts str to GBK, the type of a Python string is not GBK) decode returns a Unicode string; PS: "string" is a str instance

Coding:

Encode: Converting Unicode encoding to another encoded string

Eg:string. Encode ("GBK"), encoding Unicode string into STR,STR is GBK format encoding (misunderstanding: converting Unicode to GBK string)

Encode returns the string of STR; PS: "string" is a Unicode instance

Also change the code encoding format and the system encoding format method:

1) #-*-CODING:GBK-*-Specifies the encoding format of the code: such as GBK,UTF8, etc.

2) Reload (SYS)

Sys.setdefaultencoding (' GBK ') Specifies the default encoding format for the system (deprecated in 2.6)

1. Json

In the project, data warehousing and the reading of the Web data, to use the interface that is sufficient in the background, the server side will serialize the results of MySQL sent to the interface, interface and then deserialize the data. Both the serialization and deserialization of the database and Python have coding problems, so the design of the pre-database is consistent with the whole system, otherwise there will be a lot of coding problems. For example, the project code is GBK, the data interaction is converted into GBK, then the database encoding is the best GBK encoding. Using the JSON library in Python also specifies that the encoding is GBK, which can reduce the number of transcoding problems

JSON serialization and deserialization APIs

Import JSON

def dumps (obj, Skipkeys=false, Ensure_ascii=true, Check_circular=true,

Allow_nan=true, Cls=none, Indent=none, Separators=none,

Encoding= ' Utf-8 ', Default=none, **kw) json.loads (records, encoding)

def loads (s, Encoding=none, Cls=none, Object_hook=none, Parse_float=none,

Parse_int=none, Parse_constant=none, **kw):

"" "Deserialize" (a "str" or "Unicode" instance containing a JSON

Document) to a Python object.

If ' s ' is a ', ' str ' instance and is encoded with an ASCII based encoding

Other than utf-8 (e.g latin-1) then an appropriate ' encoding ' name

Must be specified. Encodings that is not ASCII based (such as UCS-2)

is not allowed and should is decoded to ' Unicode ' first.

1) Json.dumps, serialized into a string (eg:[[' Jimmy '],[' Helen ']); This type is STR,STR the default encoding is UTF8 (you can specify the encoding)

2) Json.load is a two-dimensional array that deserializes a string into a Pyhton array of: [U ' Jimmy '],[u ' Helen ']

3) JSON and Python serialized and deserialized data structures corresponding to

| JSON | Python |

| Object | Dict |

| Array | List |

| string | Unicode |

| Number (int) | int, Long |

| Number (real) | float |

| true | True |

| False | False |

| null | None |

1. Summary

Attention:

1) If you have special characters in the Insert database, you can use Python's third-party library mysqldb,mysqldb.escape_string (str) to automatically escape all the special characters in STR;

2) By default, Python's function entry encoding is mostly Unicode, usually we pass in STR, if the parameters are not a problem in English, Python automatically decodes str into Unicode, if there is Chinese, we need to decode Str into Unicode;

3) Multi-system, cross-platform projects, at the beginning of the design should fully consider the problem of coding, unified coding. Unified data interaction coding, unified Database encoding.

1. Resources

1) http://blog.csdn.net/zbyufei/article/details/5856730

2) http://www.python.org/

Python character encoding (python2.6)