Difference between utf8 and utf8 + BOM

One with tags, one without tags.
Bom is a byte order mark (which defines the byte sequence), because there are two types of sequence in network transmission: large and small.

Due to compatibility, UTF-8 with Bom is garbled in Some browsers.

The following information about byte order mark is searched online:
There is a character named "Zero Width no-break space" in the UCS encoding, and its encoding is feff. Fffe does not exist in the UCs, so it should not appear in actual transmission. Create a standard

We will first transmit the character "Zero Width no-break space" before transmitting the byte stream ". In this way, if the receiver receives feff, it indicates that the byte stream is big-Endian. If the receiver receives fffe, it indicates that

Byte streams are of little-Endian. Therefore, the character "Zero Width no-break space" is also called Bom.
The UTF-8 does not need BOM to indicate the byte order, but BOM can be used to indicate the encoding method. The UTF-8 code for the character "Zero Width no-break space" is ef bb bf. Therefore, if the recipient receives an EF BB BF

The byte stream at the beginning knows that this is UTF-8 encoding.
Windows uses BOM to mark the encoding of text files.

UTF-8 with Bom, all PHP cannot be identified, ef bb bf output directly, in charset = "UTF-8" page is blank, In the gb2312 page output is rare Chinese characters: kuang

 

How to convert to UTF-8 without BOM format ??
Which can be converted using ultaredit
Open your file directly with ue, and there is a conversion option in the file option
Select ASCII to uft-8.

[Utf8 + BOM generation problems and Summary]
When writing a Python script, the following problem is found: when exporting data from an xls file to a TXT file, data of the int type cannot be directly converted, the output view is found to be related to the additional information generated by the file encoding method.

File example

90905

90907

90908

90909

90939

90940

90946

90959

90961

90965

When files are encoded in ASCII, utf8, utf8, and BOM formats, the output result is as follows:

Output Using ASCII encoding:

['2014 \ r \ n', '2014 \ r \ n', '2014 \ r \ n', '2014 \ r \ n', '2014 \ r \ n ', '2017 \ r \ n', '2017 \ r \ n', '2017 \ r \ n', '2017 \ r \ n', '2017 \ r \ n ', '123']

Output Using utf8 encoding:

['2014 \ r \ n', '2014 \ r \ n', '2014 \ r \ n', '2014 \ r \ n', '2014 \ r \ n ', '2017 \ r \ n', '2017 \ r \ n', '2017 \ r \ n', '2017 \ r \ n', '2017 \ r \ n ', '123']

Output Using BOM encoding:

['\ XeF \ xbb \ xbf90905 \ r \ n', '2017 \ r \ n', '2017 \ r \ n', '2017 \ r \ n ', '2017 \ r \ n', '2017 \ r \ n', '2017 \ r \ n', '2017 \ r \ n', '2017 \ r \ n ', '123']

The original utf8 + BOM can not be directly converted to the int reason here, it in the file header inserted a file encoding information \ XeF \ xbb \ xbf, then UTF-8 (no BOM) what is the difference between the two and UTF-8?

What is Bom?

What is Bom?
Bom: byte order mark
UTF-8 Bom is also called UTF-8 signature, in fact, the BOM of UTF-8 has no effect on the UFT-8, is to support the UTF-16, UTF-32 is added

Bom indicates the encoding of the current file in the editor to facilitate editor identification. However, although Bom is not displayed in the editor, it produces output, just like an empty line.

Byte order marks are special characters at the beginning of a Unicode file to indicate whether it is big or little endian, in other words

Does the high or low order byte come first. these codes also tell whether the encoding is 8, 16 or 32 bit. you can recognize Unicode files by their starting byte order marks, and by the way Unicode-16 files are half zeroes and Unicode-32 files are three-quarters zeros. unicode endian markers

Byte-order mark description
Ef bb bf UTF-8
FF Fe UTF-16 aka UCS-2, little endian
Fe FF UTF-16 aka UCS-2, big endian
00 00 FF Fe UTF-32 aka UCS-4, little endian.
00 00 Fe FF UTF-32 aka UCS-4, big-Endian.

UTF byte order and BOM
UTF-8 uses bytes as the encoding unit, so there is no bytecode problem. The UTF-16 uses two bytes as the encoding unit. before interpreting a UTF-16 text, you must first understand the byte order of each encoding unit. For example, if the Unicode encoding of "queue" is 594e and that of "B" is 4e59. If we receive the UTF-16 byte stream "594e", is this "Kui" or "B "?

The recommended method for marking byte order in Unicode specifications is Bom. Bom is not a "bill of material" Bom, but a byte order mark. Bom is a bit clever:

There is a character named "Zero Width no-break space" in the UCS encoding, and its encoding is feff. Fffe does not exist in the UCs, so it should not appear in actual transmission. We recommend that you transmit the character "Zero Width no-break space" before transmitting the byte stream in the UCS specification ".

In this way, if the receiver receives feff, it indicates that the byte stream is big-Endian; if it receives fffe, it indicates that the byte stream is little-Endian. Therefore, the character "Zero Width no-break space" is also called Bom.

The UTF-8 does not need BOM to indicate the byte order, but BOM can be used to indicate the encoding method. The UTF-8 code for the character "Zero Width no-break space" is ef bb bf. So if the receiver receives a byte stream starting with ef bb bf, it will know that this is UTF-8 encoding.

Windows uses BOM to mark the encoding of text files.

Originally, BOM added several bytes at the beginning of the file as the mark. With this mark, some protocols and systems can be identified.

Okay, so how can we solve this problem?

How to Use BOM Header
Delete BOM Header
For the UTF-16, Python decodes BOM as an empty string. However for the UTF-8, Bom is decoded as a character, for example:

>>> Codecs. bom_utf16.decode ("UTF16") u''> codecs. bom_utf8.decode ("utf8") U' \ ufeff 'the simple practice is to use it when reading files.

Import codecs

F = codecs. Open (SYS. argv [1], 'R', 'utf _ 8_sig '). For details, see [http://docs.python.org/library/codecs.html#module-

Encodings. utf_8_sig | http://docs.python.org/library/codecs.html?module-encodings.utf_8_sig]

Or:

Add U. lstrip (Unicode (codecs. bom_utf8, "utf8") BOM Header
Out = file ("somefile", "W ")
Out. Write (codecs. bom_utf8)
Out. Write (unicodestring. encode ("UTF-8 "))
Out. close () out = file ("somefile", "W") out. write (codecs. bom_utf8) out. write (unicodestring. encode ("UTF-8") out. detailed process description of close () can be

See [http://mindprod.com/jgloss/encoding.html | http://mindprod.com/jgloss/encoding.html]

References:

[Http://blog.sina.com.cn/s/blog_3e9d2b350100as0b.html | http://blog.sina.com.cn/s/blog_3e9d2b350100as0b.html]

[Http://4nail.iteye.com/blog/840612 | http://4nail.iteye.com/blog/840612]

[Which of the following is the common use of utf8, utf8, and BOM webpage code ?]

First, what is Bom. This is not explained. It is detailed on Wikipedia. Http://en.wikipedia.org/wiki/byte_order_mark ....

Using Bom on a webpage is an error. Bom is not designed to support HTML and XML. To identify text encoding, HTML has the charset attribute, XML has the encoding attribute, and there is no need to pull BOM to support scenes. Although theoretically BOM can be used to identify HTML pages of UTF-16 code, few people do this in actual engineering. After all, this encoding of UTF-16 even ASCII are dubyte, it is not applicable to do web pages.

In fact, Bom is not a bad habit. Bom is also part of the Unicode standard and has its specific applicability. Usually bomis used to mark the unicodepure character stream, used to identify a convenient character processing program reading the. txt file which is Unicode encoding (UTF-8, UTF-16BE, UTF-16LE ). Windows processes BOM better because it integrates Unicode recognition codes into APIs, mainly createfile (). When a text file is opened, it automatically identifies and removes the BOM. This is a historical reason for using Windows because it was originally originated from a multi-page environment. When Unicode is introduced, Windows designers hope to be able to be compatible with Unicode and non-Unicode (multiple byte) text files without your attention, so they can only use this small trick. In contrast, Linux systems such as Linux have a short deployment time in Multi-locale environments. In addition, the Community itself has enough power to move forward with light load (spof: microsoft's requirements for compatibility are indeed very paranoid, and any practices that undermine compatibility are not allowed, so many times they are tied to their own hands ), so dry brittle step into the UTF-8. Of course, there is a transitional period in the middle, such as from the initial full UTF-8 of GTK + 2.0 released to basically all GTK developers are not using multiple locale GTK + 1.2, I have been there for at least three to four years.

Bom is not popular in UNIX environments, because many UNIX programs do not bird Bom. The main problem lies in the first line of all the scripting languages of UNIX #! This depends on Shell parsing. Many shells do not check BOM for compatibility reasons. Therefore, when adding Bom, shell will interpret it as a common character input, causing damage #! Mark, this is troublesome. In fact, many modern scripting languages, such as Python, can process BOM in their interpreters themselves, but shell is stuck here, there is no way, you can only lie down and shot. This cannot be blamed on shell, because BOM itself violates a Common Unix design principle, that is, the data in the document must be visible. Bom cannot be edited as visible characters in the text editor, which is not satisfactory to many UNIX developers.

By the way, even if the script language can process Bom, using BOM everywhere is not recommended. Each scripting language has its own set of Unicode processing. Python #-*-coding: UTF-8-*-And Perl's use utf8 are simpler and more reliable than Bom. Another good news is that even friends who have to switch between windows and UNIX will not be miserable. Thanks to the Unix environment, we also have the vim artifact. Even in the case of BOM barrier, we can solve the problem by running the set nobomb; Set fileencoding = utf8; W command.

In the end, it seems that only Windows insist on BOM.

P.s.: Why does Vim remove bomb in UNIX. Because Vim has a strange bug in Windows, it always recognizes the UTF-16 file as a binary file, while Vim in Unix (either Linux or Mac) is fine. This problem has been followed by me from Vim 6.8 to VIM 7.3. It is unclear whether this is a Vim bug or a bug in my own. vimrc file.

[Here is a brief description of Unicode encoding. It briefly describes the terminologies such as UCOS, UTF, BMP, and Bom]
This is an interesting book written by programmers. The so-called fun refers to the ability to easily understand some previously unclear concepts and enhance knowledge, similar to upgrading RPG games. There are two reasons for organizing this article:

Question 1:
Using the Save As in Windows notepad, you can convert between GBK, Unicode, Unicode big endian, and UTF-8 encoding methods. It is also a TXT file. How does Windows identify the encoding method?

I found that Unicode, Unicode big endian, and UTF-8-encoded TXT files start with several more bytes, namely ff, Fe (UNICODE ), fe and FF (UNICODE big endian), EF, BB, BF (UTF-8 ). But what standards are these tags based on?

Question 2:
A convertutf. C was recently seen online, implementing mutual conversion between UTF-32, UTF-16, and UTF-8. I used to understand Unicode (ucs2), GBK, and UTF-8 encoding methods. But this program makes me a little confused, don't remember what the UTF-16 and ucs2 has.
After checking relevant information, I finally figured out these problems and learned some Unicode details. Write an article and send it to friends with similar questions. This article tries its best to be easy to understand when writing, but requires readers to know what is byte and what is hexadecimal.

0, big endian, and little endian
Big endian and little endian are different ways for CPUs to process the number of multi-word segments. For example, the Unicode code of the Chinese character is 6c49. When I write a file, do I write 6C in front or 49 in front? If you write 6C in front, it is big endian. Write 49 in front, that is, little endian.

The word "endian" comes from Gulliver Travel Notes. The civil war in the minor people's country originated from the fact that the big-Endian attack or the little-Endian attack were initiated when the eggs were eaten. As a result, there were six rebels, one of the emperors gave life and the other lost the throne.

We generally translate endian into byte order, and call Big endian and little endian "Big tail" and "Small Tail ".

1. character encoding and inner code. This section introduces Chinese character encoding.
The character must be encoded before it can be processed by the computer. The default encoding method used by the computer is the computer's internal code. Early computers used 7-bit ASCII code. To process Chinese characters, programmers designed gb2312 for simplified Chinese and big5 for traditional Chinese.

Gb2312 (1980) contains a total of 7445 characters, including 6763 Chinese characters and 682 other symbols. The inner code range of the Chinese character area is high byte from the B0-F7, low byte from the A1-FE, the occupied bitwise of the Code is 72*94 = 6768. Five of them are D7FA-D7FE.

Gb2312 supports too few Chinese characters. The Chinese character extension specification gbk1.0 in 1995 contains 21886 characters, which are divided into Chinese Character areas and graphic symbol areas. The Chinese Character area contains 21003 characters. In 2000, gb18030 was replaced

Gbk1.0 official national standard. The standard includes 27484 Chinese characters, as well as Tibetan, Mongolian, and Uyghur texts. The current PC platform must support gb18030, which is not required for embedded products. Therefore, mobile phones and MP3 generally only support gb2312.

From ASCII, gb2312, GBK to gb18030, these encoding methods are backward compatible, that is, the same character always has the same encoding in these schemes, and the following standard supports more characters. In these encodings, both English and Chinese can be processed in a unified manner. The difference between Chinese encoding is that the maximum bit of a high byte is not 0. According to programmers, gb2312, GBK, and gb18030 are both dual-byte character sets (DBCS ).

In some Chinese Windows, the default internal code is GBK. You can use the gb18030 upgrade package to upgrade to gb18030. However, it is difficult for ordinary people to use the characters added by gb18030 to GBK. We usually use GBK to refer to the Chinese Windows internal code.

Here are some details:

The original gb2312 text is still a location code. From the location code to the inner code, you need to add A0 to the high byte and low byte respectively.

In DBCS, the storage format of GB internal code is always big endian, that is, the top is in front.

The maximum bits of two gb2312 bytes is 1. However, only 128*128 = 16384 digits are allowed. Therefore, the highest bit of the low byte of GBK and gb18030 may not be 1. However, this does not affect the DBCS streams.

Resolution: when reading the DBCS upstream stream, the next two bytes can be used as a dual-byte encoding when the high byte is 1, regardless of what the high byte is.

2. Unicode, UCOS, and UTF previously mentioned that the encoding methods from ASCII, gb2312, GBK to gb18030 are backward compatible. Unicode is only compatible with ASCII (more accurately, it is compatible with iso-8859-1) and is not compatible with the GB code. For example, the Unicode code of the Chinese character is 6c49, And the GB code is Baba.

Unicode is also a method of character encoding, but it is designed by international organizations to accommodate all the languages and texts in the world. The Unicode name is "Universal multiple-octet coded character set", which is short for UCOS. UCOS can be seen as the abbreviation of "Unicode Character Set.

According to Wikipedia (http://zh.wikipedia.org/wiki/), there have been two organizations in history attempting to design Unicode independently, namely the International Organization for Standardization (ISO) and the Association of a software manufacturer (unicode.org ). ISO has developed the ISO 10646 project, and the Unicode Association has developed the Unicode project.

Around 1991, both parties realized that the world does not need two incompatible character sets. As a result, they began to merge their work results and work together to create a single coding table. Since unicode2.0, the Unicode project adopts the same font and character code as ISO 10646-1.

Currently, both projects still exist and their respective standards are published independently. The latest version of Unicode Association is Unicode 4.1.0 in 2005. The latest ISO standard is 10646.

UCOS specifies how to use multiple bytes to represent various texts. How these encodings are transmitted is stipulated by the UTF (UCS Transformation Format) specification, and common UTF specifications include UTF-8, UTF-7, and UTF-16.

IETF rfc2781 and rfc3629 with RFC consistent style, clear, bright and rigorous description of the UTF-16 and UTF-8 coding method. I cannot remember that IETF is short for Internet Engineering Task Force. However, the RFC maintained by IETF is the basis of all regulations on the Internet.

3, UCS-2, UCS-4, bmp ucs has two formats: UCS-2 and UCS-4. As the name suggests, UCS-2 is to use two bytes of encoding, UCS-4 is to use 4 bytes (actually only 31 bits, the highest bit must be 0) encoding. Let's make some simple mathematical games:

UCS-2 has 2 ^ 16 = 65536 bits, UCS-4 has 2 ^ 31 = 2147483648 bits.

Ucs-4 is divided into 2 ^ 7 = 128 groups based on the highest byte with the highest bit of 0. Each group is further divided into 256 Plane Based on the next high byte. Each plane is divided into 3rd rows (rows) based on 256 bytes, and each row contains 256 cells. Of course, cells in the same row are only different from the last byte, and the rest are the same.

The plane 0 of group 0 is called Basic multilingual plane, that is, BMP. Or in the UCS-4, the code bit with the height of two bytes 0 is called BMP.

Remove the bmp of the UCS-4 from the first two zero bytes to get the UCS-2. Add two zero bytes before the two bytes of the UCS-2 to get the bmp of the UCS-4. Currently, the UCS-4 specification does not contain any characters other than BMP.

4. UTF Encoding

The UTF-8 is coded in 8 bits. The encoding from UCS-2 to UTF-8 is as follows:

UCS-2 encoding (HEX) UTF-8 byte stream (Binary)
0000-007f 0 xxxxxxx
0080-07ff 110 XXXXX 10 xxxxxx
0800-FFFF 1110 XXXX 10 xxxxxx 10 xxxxxx

For example, the Unicode code of the Chinese character is 6c49. 6c49 is between 0800-ffff, so it must use a 3-byte template: 1110 XXXX 10 xxxxxx 10 xxxxxx. Write 6c49 as binary: 0110 110001 001001. Use this bit stream to replace X in the template. The result is 11100110 10110001 10001001, that is, E6 B1 89.

Readers can use NotePad to test whether our encoding is correct.

UTF-16 is encoded in 16 bits. The UTF-16 code is equal to the 16-bit unsigned integer corresponding to the UCS code for a UCS code that is less than 0x10000. An algorithm is defined for the UCS code not less than 0x10000. However, because the actual use of ucs2, or ucs4 BMP must be less than 0x10000, so for now, it can be considered that the UTF-16 and UCS-2 are basically the same. But UCS-2 is only a coding scheme, UTF-16 is used for actual transmission, so we have to consider the problem of byte order.

5. UTF byte order and BOM
UTF-8 uses bytes as the encoding unit, so there is no bytecode problem. The UTF-16 uses two bytes as the encoding unit. before interpreting a UTF-16 text, you must first understand the byte order of each encoding unit. For example, if the Unicode encoding of "queue" is 594e and that of "B" is 4e59. If we receive the UTF-16 byte stream "594e", is this "Kui" or "B "?

The recommended method for marking byte order in Unicode specifications is Bom. Bom is not a "bill of material" Bom, but a byte order mark. Bom is a bit clever:

There is a character named "Zero Width no-break space" in the UCS encoding, and its encoding is feff. Fffe does not exist in the UCs, so it should not appear in actual transmission. We recommend that you transmit the character "Zero Width no-break space" before transmitting the byte stream in the UCS specification ". In this way, if the receiver receives feff, it indicates that the byte stream is big-Endian; if it receives fffe, it indicates that the byte stream is little-Endian. Therefore, the word "Zero Width no-break space" is also called Bom.

The UTF-8 does not need BOM to indicate the byte order, but BOM can be used to indicate the encoding method. The UTF-8 code for the character "Zero Width no-break space" is ef bb bf (the reader can verify it with the encoding method we described earlier ). So if the receiver receives a byte stream starting with ef bb bf, it will know that this is UTF-8 encoding.

Windows uses BOM to mark the encoding of text files.

6. Further references
This article mainly references "Short overview of ISO-IEC 10646 and Unicode" (http://www.nada.kth.se/i18n/ucs/unicode-iso10646-oview.html ).

Two seemingly good documents:

"Understanding Unicode a general introduction to the Unicode Standard" (http://scripts.sil.org/cms/scripts/page.php? Site_id = nrsi & item_id = IWS-Chapter04a)
"Character set encoding basics understanding Character Set encodings and legacy encodings" (http://scripts.sil.org/cms/scripts/page.php? Site_id = nrsi & item_id = IWS-Chapter03)
There should be UTF-8, UCS-2, GBK software packages for mutual conversion on the web, including versions that use Windows APIs and that do not use Windows APIs.

Difference between utf8 and utf8 + BOM