Python source code file Text Encoding

Source codeThe encoding problem exists for the source files of any language, but this problem is more prominent for the script language.

Some may say that since the sourceCodeIt can be normally displayed in the text editor, indicating that there is no problem with encoding. The editor can identify it. Why does Python still require declaring the source code encoding?

This is because the explicit declaration of encoding can simplify the implementation of the python interpreter, so that it does not guess the encoding of the source file. This will increase the complexity of the interpreter and slow downProgramThe execution speed (after all, explain the execution ). Moreover, the program must be accurate and cannot be guessed.

From Python 2.3, You can explicitly declare character encoding in the Python source file. The default value is 7-bit ASCII encoding.

The character encoding declaration is implemented by a magic comment in the first or second line of the source file:

# Coding = <encoding name>

As to why it must be the first or second line, this is a good explanation: to explain a file, you must first know the character encoding, but on Unix/Linux platforms, the first line may be left with the python interpreter declaration ,#! /Usr/bin/python, which is also a magic annotation.

If you do not know the declared Python interpreter, an error will be reported during the compilation phase (for cpython, this refers to converting the py file to the PyC file.

If the declared encoding is inconsistent with the actual encoding (that is, the file is actually saved in another encoding), there is a high possibility of errors, but it is not always wrong, after all, there may be some overlapping codes.

If you do not declare the source file encoding, but use any character other than 7-bit ASCII, compile the file (for cpython, this refers to converting the py file to the PyC file) and displays syntaxerror.

Most non-7-bit ASCII characters appear in string literal constants. Some people will think that using Unicode string literal constants will solve the encoding problem, but this is completely another problem, if the source file encoding is not correct, the Unicode string cannot be discussed at all.

For Windows platforms, if the source file is stored in UTF-8 format, it will add BOM (byte order mark) at the beginning of the file. For UTF-8, this is \ XeF \ xbb \ xbf, at this time, you do not need to declare the file encoding. If you want to declare the file, it must also be UTF-8; otherwise, the compilation fails.

The interactive input window contains the source file encoding, which allows Chinese Input, obviously not ASCII.

The line feed sequence for Windows and Unix/Linux is different, which may also cause problems.

In short, if there is a problem with encoding, there will be errors during compilation. If the output is garbled during runtime, it should be because the Unicode string is not used or the output window is not supported.

Reference:

Http://www.python.org/dev/peps/pep-0263/

Http://bugs.python.org/issue526840

These two documents describe implementation-related details.