Detection and deletion of BOM in UTF-8 coding

The so-called BOM is the full name of ByteOrderMark, which is a Unicode character, usually at the beginning of the text, used to identify the byte order (Big/LittleEndian ), it can also identify the encoding (UTF-8/16/32), if it appears in the middle of the text, it is interpreted as zerowidthn... "> <LINKhref =" http://www.php

 

The so-called BOM is the Byte Order Mark, which is a Unicode character that usually appears at the beginning of the text to identify the Byte Order (Big/Little Endian ), in addition, the identifier code (UTF-8/16/32) can be interpreted as zero width no-break space if it appears in the middle of the text. Note: For more information about Unicode, see UTF-8, UTF-16, UTF-32 & BOM. For UTF-8/16/32, 8/16/32 in their names refer to the number of bits in the encoding unit, that is, their encoding units are 8/16/32 bits, to convert to byte is 1/2/4 bytes, if it is multi-byte, it will involve the byte order, the UTF-8 in a single byte as the encoding unit, so there is no byte order. The main advantage of UTF-8 is that it can be compatible with ASCII, but if you use BOM, this advantage is gone, in addition, the existence of BOM may also lead to some problems, for example, the following errors may be caused by BOM:

• Shell: No such file or directory
• PHP: Warning: Cannot modify header information-headers already sent

Before discussing in detail the problem of BOM detection and deletion in UTF-8 coding, we may try to warm up with an example:

Shell> curl-s http://phone.10086.cn/| head-1 | sed-n l \ 357 \ 273 \ 277\ R$

As shown above, the first three bytes are 357, 273, and 277, which are the BOM of the octal component.

Shell> curl-s http://phone.10086.cn/| head-1 | hexdump-C00000000 ef bb bf 3c 21 44 4f 43 54 59 50 45 20 68 74 6d |...... |

As shown above, the first three bytes are EF, BB, and BF, which are the hexadecimal BOM. Note: When a third-party website page is used, examples cannot be always available. In actual project development, may face hundreds of thousands of text files, if there are a few files mixed into the BOM, it is difficult to notice, if there is no UTF-8 text file with BOM, you can use vi to write several articles. The related commands are as follows:

Set UTF-8 encoding:

: Sets fileencoding = UTF-8

Add BOM:

: Set bomb

Delete BOM:

: Set nobomb

Query BOM:

: Set bomb?

How to check BOM in UTF-8 coding?

Shell> grep-r-I-l $ '^ \ xEF \ xBB \ xBF'/path

How do I delete BOM from UTF-8 encoding?

Shell> grep-r-I-l $ '^ \ xEF \ xBB \ xBF'/path | xargs sed-I's/^ \ xEF \ xBB \ xBF //; q'

Recommendation: If you use SVN, you can add relevant code to the pre-commit hook to prevent BOM.

#! /Bin/bashREPOS = "$1" TXN = "$2" SVNLOOK =/usr/bin/svnlookfor FILE in $ ($ SVNLOOK changed-t "$ TXN" "$ REPOS" | awk '/^ [AU]/{print $ NF }'); do if $ SVNLOOK cat-t "$ TXN" "$ REPOS" "$ FILE" | grep-q $ '^ \ xEF \ xBB \ xBF '; then echo "Byte Order Mark be found in $ FILE" 1> & 2 exit 1 fidone

Many shell commands are used in this article, which are limited in length and will not be described in detail. If you do not understand them, please search for them by yourself.

PS: Some other command lines:

 

1	Awk '{if (NR = 1) sub (/^ \ xef \ xbb \ xbf/, ""); print}' text.txt

2

3	Tail -- bytes = + 4 text.txt

4

5	Tail-c + 4 UTF8> UTF8.nobom

 

 

Two other cases of abnormal python UTF-8 BOM encoding:

Http://hi.baidu.com/leejun_2005/item/bf28ded296cb8ebf33db9096

Http://hi.baidu.com/leejun_2005/item/8c8855757225835b0d0a0780