UTF-8 encoded files in processing should pay attention to the BOM file header problems

Recently, when developing a common Java-based XML parser for the project team, a method was designed to read out-of-the-box XML files for analysis and processing, although XML was encoded using UTF-8. However, after writing a test UTF-8 XML file with UltraEdit, the program has an error reading the file:

Parse Fatal Error at line 1 column 1: Content is not allowed in the preface. Org.xml.sax.SAXParseException:Content isn't allowed in Prolog ....

Repeatedly checking the code can not be a problem, the XML file is also correct, try to try to think of the solution. So with the idea editor wrote a same XML file, the program runs normally. Find out on the internet, inadvertently found UTF-8 code actually has a so-called BOM file head problem. So, what is the BOM?

The BOM, byte order mark, is the byte order mark. There is a character called "ZERO WIDTH no-break SPACE" in the UCS encoding, and its encoding is FEFF. FFFE is not a character in UCS, so it should not appear in the actual transmission. The UCS specification recommends that we transmit the character "ZERO WIDTH No-break SPACE" before transmitting the byte stream, so that if the recipient receives FEFF, it indicates that the byte stream is Big-endian; If you receive FFFE, it indicates that the byte stream is Little-endian's. So the character "ZERO WIDTH no-break SPACE" is also called a BOM.

The UTF-8 itself does not require a BOM to indicate the byte order, but it can be used to indicate the encoding using a BOM. The UTF-8 code for the character "ZERO WIDTH no-break SPACE" is the EF BB BF. So if the receiver receives a byte stream beginning with the EF BB BF, it knows that this is UTF-8 encoded. Therefore, some editors will automatically add the EF BB BF three bytes to the file header when creating and saving the UTF-8 encoded XML file, which is used to make a BOM. UltraEdit The default is this way, clever instead of trouble! The idea editor created the generated UTF-8 encoded XML file without this BOM file header, so the problem phenomenon described at the beginning of this article appears.

Later studied the UltraEdit, found that it provides a save option, in the first save or later "Save as", choose to Save as UTF-8 without BOM can be.

Now that I have found the reason, I think I can not let various editors to accommodate my program, or let my program to accommodate them, that is, in the program to increase the ability to automatically identify the BOM file header. In this way, whether or not with the BOM file header UTF-8 encoded XML file, my program can be compatible with the use of, in fact, the realization of this function was found, very simple, did not increase too much work, why not?

UTF-8 encoded files in processing should pay attention to the BOM file header problems