Python reads the file, and when split is used, the \ufeff appears

1. Questions

When you use python3.6 to read a file, the file contents are similar to the following as normal routines:

Ah Ah Ah | | Aaa

No no no No | | Bbb

when using Utf-8 to read a document and split with the split function, it is found that the first line of characters is split, and a \ufeff is more inexplicable.

For this reason, an error occurs when the first row of data is involved in subsequent use of the = = or the x in list operation .

Debug the first line see (the second line is no problem after)

2. Causes

Reference 1:

The Unicode character U+feff is the byte order mark, or BOM, and are used to tell the difference between Big-and Little-en Dian UTF-16 encoding

U=U' ABC 'E8=U.Encode(' Utf-8 ') # encode without BOM (BOM not considered)E8s=U.Encode(' Utf-8-sig ')  # encode with BOM (considering BOM) e16 = U. ( ' utf-16 '  # Encode with Bome16le = U. ( ' utf-16le '  # Encode without Bome16be = U. ( ' utf-16be '  # Encode without BOM               

Reference 2

\ufeff is a special logo that indicates the encoding method,

the sequence of bytes, which is the order of the bytes, refers to the order in which multibyte data is stored in memory, and on almost all machines, multibyte objects are stored as contiguous sequences of bytes, based on the order in which the information is stored in contiguous memory, and the byte order is divided into Big Endian. and Small-endian (Little Endian) two classes. (then there are two big CPU factions involved: The PowerPC 970 processor uses the big endian to store data, and a processor such as the x86 series stores data in little endian mode). The big-endian and small-order explanations are as follows:

• The Big Endian is the low address side that holds the high byte.
• Little Endian refers to low-address-side storage of lower bytes.

For its role and more, see reference

3. Workaround

One way:

The string is encode into a byte array by Utf-8, and then the byte array is decode using UTF-8-SIG, namely:

Templatelist = []     forLineinchOpen'TemplateResult.txt', encoding='Utf-8'): Tmps= Line.strip (). Split ('|') Templatelist.append (Tmps[0].encode ('Utf-8'). Decode ('Utf-8-sig'). Strip (). Replace ('. ',"'))

Another way to open a file directly using ' Utf-8-sig ':

    Templatelist = []    for in open ('templateResult.txt', encoding='utf-8-sig'):        = Line.strip (). Split ('| ' )        Templatelist.append (Tmps[0].strip (). replace ('.  "))

　　

4. Summary

Writing a document or reading a document is a common practice used by Python, such as opening a document using open (' Test.txt ', encoding= ' Utf-8 '), which may cause problems when processing the first row of data because of its own neglect.

In this paper, the causes of errors and solutions are described.

Reference :

Https://stackoverflow.com/questions/17912307/u-ufeff-in-python-string

https://songlee24.github.io/2015/05/02/endianess/

Python reads the file, and when split is used, the \ufeff appears