Python's annoying copy (\ xef \ xbf \ xbd) and xefxbf

Python's annoying copy (\ xef \ xbf \ xbd) and xefxbf

First, you must know what \ xef \ xbf \ xbd is.

>>> U' \ uFFFD '. encode ('utf-8')' \ xef \ xbf \ xbd'

From this we can know that \ xef \ xbf \ xbd is utf8 encoded '\ uFFFD', so what is this '\ uFFFD?

It turns out that there must be some words in the conversion process of Unicode and the old encoding system, which cannot be expressed by Unicode. Unicode officially uses a placeholder to represent these words, which is: U + fffd replacement character.

>>> "\ Xef \ xbf \ xbd". decode ("UTF-8") U' \ ufffd'

Therefore, if some characters in your text cannot be represented by unicode, it will be converted to \ xef \ xbf \ xbd when the python program reads it. If unicode is changed in some places, in this case

U' \ ufffd ', but you can no longer convert it to str.

If you happen to use these strings as Dictionary keys, a key_error error will be reported.