After UTF-8 is converted to GBK, the question mark "?" appears "?" Cause

The question mark "? "Reason

Webpage encoding conversion is the first step for parsing and processing webpages.

More and more web pages on the Internet use UTF-8 encoding, UTF-8 with 1 to 6 bytes encoded Unicode characters, contains more than 0.1 million characters, BMP part also has more than 60 thousand characters

 

During encoding conversion, we often need to convert to GBK encoding for subsequent processing. After conversion, many webpages will find a large number of consecutive question marks :????????

These disgusting question marks are introduced in the encoding conversion phase because:

 

The GBK character set contains more than 20 thousand characters, much less than the number of characters in the UTF-8.

When converting to GBK encoding, the encoding will fall beyond the GBK character set and cannot be converted to GBK encoding. These characters are converted '? '

 

UTF-8: Extended bytes (1 ASCII, 2 Greek letters and typographical characters, 3 Chinese characters and other multi-byte East Asian language, 4-plane symbols and special symbols, etc ),

Some of the double byte characters are not in the GBK character set, usually the characters that cannot be recognized by the UTF-8 are very uncommon, almost difficult to encounter, do not consider; but there is a character very special:C2a0

 

C2a0 is the formatting space in utf8 (different from the space of ASI = 20). This special character (UNICODE serial number is 0xa0) is not in the GBK character set. But it is frequently used in XML, HTML, and other formats. A large number of UTF-encoded web pages use this character as a placeholder space. In addition, different browsers use different processing methods: IE recognizes the symbol and displays it with spaces. Firefox replaces it with the XML Escape Character & nbsp; when the webpage uses c2a0 for text formatting, when we convert the webpage encoding to GBK, there will be many "? "Question mark

 

A simple solution is to filter webpages before transcoding:

If (code_type = "utf8 ")

{

STR = Str. Replace ('c2a0 ','');

}