Python | Multiple encoding Files (Chinese) garbled problem solving

Problem clue

1. 1

   As you can tell, the default encoding for a text file is not UTF8.

   We open a text file and click Save As

2. 2

   We see that the default encoding is ANSI in the Code column of the new window. Regardless of what the encoding is, but through the drop-down list we know that this encoding is not UTF8.

   END

Coding test

1. 1

   For the Chinese display in Python, we often use the UTF8 and GBK codes. For these two kinds of coding I do not introduce, in short, are specifically able to deal with the Chinese encoding method.

2. 2

   We first tested the GBK decoding of the text file. We found that the text file content of this encoding can be displayed normally, but using UTF8 decoding, program error, throw Decodeerror exception

3. 3

   Similarly, we tested the UTF8 decoding of the Python script file. We found that the text file content under this encoding can be displayed normally, but the Chinese part is garbled using GBK decoding

4. 4

   Test description, for the text file needs to use GBK decoding, and for the script file needs UTF8 decoding, that is, the text file is GBK encoded, and the script is UTF8

   END

Settlement method One: Exception handling

• We found from the above code test that the text file throws an exception when using UTF8 decoding, so we can do the following in the code-that is, the GBK decoding when the exception is thrown.

• After testing, the discovery program can meet the normal display of two files

  END

Resolution II: File types

• Because the two files are determined, you can directly determine the file type selection corresponding decoding. It has been tested and has been successful.

  END

Ultimate Solution: Chardet

1. The Chardet module detects character encodings and should be the ultimate solution to similar problems. Install it first.

2. Test the encoding of both files using the code below. We see that a dictionary is returned through the Chardet module. The first element of the dictionary is the probability of the encoding check, and the latter is the encoding type

3. We see that the encoding of the text file is GB2312, and we use the GBK decoding above is also possible, it is because gb2312 is a subset of GBK (GBK/1, GBK/2 is the GB2312 region)

4. Now let's revise the final code to get rid of this coding problem ~

Python | Multiple encoding Files (Chinese) garbled problem solving