Universalchardet, an automatic character set detection class with higher accuracy than IE

There are many languages in the world that, in order to be able to express these languages on the computer, various countries have developed their own different character encoding standards before the emergence of UTF-8. They are not compatible with each other. It was not until the emergence of UTF-8 that there was a unified coding standard (now found that UTF-8 does not include all the words in the world), but the old standards are still in their respective use. For example, there are still many gb2312 webpages in China.

How can we identify the encoding used by a webpage?

First, the webpage or server directly reports the browser. What encoding is used for this page. For example, the Content-Type attribute of the HTTP header and the charset attribute of the page. This is easy to implement. You only need to detect these attributes to know what encoding is used.

Second, the browser automatically guesses. This is similar to artificial intelligence. For example, if some webpages do not write the charset attribute, We will manually select the page encoding when we see garbled characters on the page. If we find the code is garbled, we will change it until the display is normal.

Today's articleArticleThe second method is used.ProgramEnable Automatic guessing of character sets used by pages or files.

The principle is to analyze the characteristics of the characters in statistics to determine which characters are the most common. Mozilla has a special article titled a composite approach to language/encoding detection. One of the authors is shanjian Li, a Beijinger from China.

Okay, specificCodeIn fact, Mozilla has already implemented it using C ++, and its name is universalchardet. However, I have searched the internet and cannot find the. NET implementation class library. Only Google Code has Java translation code. No way. Translate the code into C.

C # implementedSource code: Http://code.google.com/p/nuniversalchardet/

 

Ps1. By the way, why is the title more accurate than IE? That's because IE also comes with the character set guessing function, some people have implemented the function library (http://www.codeproject.com/KB/recipes/DetectEncoding.aspx) by calling the IE interface to guess the character set, but I tried, the accuracy of this interface is not high, the chance of success is much lower than universalchardet.

Ps2. nchardet is widely circulated on the Internet. It is implemented based on C # Of chardet in earlier versions of Mozilla. The accuracy is also relatively low, which is similar to the interface success rate of IE.

PS3. references

Juniversalchardet: http://code.google.com/p/juniversalchardet/ (Java code has bugs in the big5prober and gb18030prober classes, C # has been fixed)

principle reference: http://www.mozilla.org/projects/intl/UniversalCharsetDetection.html