Php evaluate several methods for dynamic conversion from gb encoding to UTF-8 encoding

Php evaluates several methods of dynamic conversion from gb encoding to UTF-8 encoding, and reads several methods of php's dynamic conversion from gb encoding to UTF-8 encoding, the article "IP address-Geographic location conversion evaluation" mentions that using the ip2addr function to directly read IP database files is the most efficient. compared with using the MySQL database to store IP data, SQL queries are the most efficient. However, the IP database file "> <LINKhref =" ht

The article "IP address-> geographic location conversion evaluation" mentions that using the ip2addr function to directly read IP database files is the most efficient. compared with using the MySQL database to store IP data, SQL queries are the most efficient. However, the IP database file QQWry. dat is GB2312 encoded. Now I need the geographic location results for UTF-8 encoding. If you use the MySQL method, you can convert the data into UTF-8 encoding when it is stored in the database, once and for all. However, the QQWry. dat file cannot be modified, and the output result of the ip2addr function can only be dynamically converted.

There are at least four methods for dynamic conversion of GB-> UTF-8 encoding:

Extended conversion using PHP iconv

Extended conversion using PHP mb_string

Use swap table conversion, swap table stored in MySQL database

Convert tables with swap tables and store them in text files.

The first two methods can be used only when the server has made corresponding settings (the corresponding extensions are compiled and installed. My VM does not have these two extensions, so I have to consider the last two methods. The first two methods are not evaluated in this document.

The evaluation procedure is as follows (for func_ip.php, see the article "IP address-> geographic location conversion evaluation ):

Require_once ("func_ip.php ");
Function u2utf8 ($ c ){
$ Str = "";
If ($ c <0x80 ){
$ Str. = $ c;
} Elseif ($ c <0x800 ){
$ Str. = chr (0xC0 | $ c> 6 );
$ Str. = chr (0x80 | $ c & 0x3F );
} Elseif ($ c <0x10000 ){
$ Str. = chr (0xE0 | $ c> 12 );
$ Str. = chr (0x80 | $ c> 6 & 0x3F );
$ Str. = chr (0x80 | $ c & 0x3F );
} Elseif ($ c <0x200000 ){
$ Str. = chr (0xF0 | $ c> 18 );
$ Str. = chr (0x80 | $ c> 12 & 0x3F );
$ Str. = chr (0x80 | $ c> 6 & 0x3F );
$ Str. = chr (0x80 | $ c & 0x3F );
}
Return $ str;
}
Function GB2UTF8_ SQL ($ strGB ){
If (! Trim ($ strGB) return $ strGB;
$ StrRet = "";
$ IntLen = strlen ($ strGB );
For ($ I = 0; $ I <$ intLen; $ I ++ ){
If (ord ($ strGB {$ I})> 127 ){
$ StrCurr = substr ($ strGB, $ I, 2 );
$ IntGB = hexdec (bin2hex ($ strCurr)-0x8080;
$ StrSql = "SELECT code_unicode FROM nnstats_gb_unicode
WHERE code_gb = ". $ intGB." LIMIT 1"
;
$ ResResult = mysql_query ($ strSql );
If ($ arrCode = mysql_fetch_array ($ resResult) $ strRet. = u2utf8 ($ arrCode ["code_unicode"]);
Else $ strRet. = "?? ";
$ I ++;
} Else {
$ StrRet. = $ strGB {$ I };
}
}
Return $ strRet;
}
Function GB2UTF8_FILE ($ strGB ){
If (! Trim ($ strGB) return $ strGB;
$ ArrLines = file ("gb_unicode.txt ");
Foreach ($ arrLines as $ strLine ){
$ ArrCodeTable [hexdec (substr ($ strLine, 0, 6)] = hexdec (substr ($ strLine, 7, 6 ));
}
$ StrRet = "";
$ IntLen = strlen ($ strGB );
For ($ I = 0; $ I <$ intLen; $ I ++ ){
If (ord ($ strGB {$ I})> 127 ){
$ StrCurr = substr ($ strGB, $ I, 2 );
$ IntGB = hexdec (bin2hex ($ strCurr)-0x8080;
If ($ arrCodeTable [$ intGB]) $ strRet. = u2utf8 ($ arrCodeTable [$ intGB]);
Else $ strRet. = "?? ";
$ I ++;
} Else {
$ StrRet. = $ strGB {$ I };
}
}
Return $ strRet;
}
Function EncodeIp ($ strDotquadIp ){
$ ArrIpSep = explode ('.', $ strDotquadIp );
If (count ($ arrIpSep )! = 4) return 0;
$ IntIp = 0;
Foreach ($ arrIpSep as $ k => $ v) $ intIp + = (int) $ v * pow (256, 3-$ k );
Return $ intIp;
// Return sprintf ('% 02x % 02x % 02x % 02x', $ arrIpSep [0], $ arrIpSep [1], $ arrIpSep [2], $ arrIpSep [3]);
}
Function GetMicroTime (){
List ($ msec, $ sec) = explode ("", microtime ());
Return (double) $ msec + (double) $ sec );
}
For ($ I = 0; $ I <100; $ I ++) {// 100 IP addresses are randomly generated.
$ StrIp = mt_rand (0,255). ".". mt_rand (0,255). ".". mt_rand (0,255). ".". mt_rand (0,255 );
$ ArrAddr [$ I] = ip2addr (EncodeIp ($ strIp ));
}
$ ResConn = mysql_connect ("localhost", "netnest", "netnest ");
Mysql_select_db ("test ");
// Evaluate the encoding conversion of MySQL queries
$ DblTimeStart = GetMicroTime ();
For ($ I = 0; I I <100; $ I ++ ){
$ StrUTF8Region = GB2UTF8_ SQL ($ arrAddr [$ I] ["region"]);
$ StrUTF8Address = GB2UTF8_ SQL ($ arrAddr [$ I] ["address"]);
}
$ DblTimeDuration = GetMicroTime ()-$ dblTimeStart;
// The evaluation is complete and the result is output
Echo $ dblTimeDuration; echo "\ r \ n ";
// Evaluate the encoding conversion of text file queries
$ DblTimeStart = GetMicroTime ();
For ($ I = 0; I I <100; $ I ++ ){
$ StrUTF8Region = GB2UTF8_FILE ($ arrAddr [$ I] ["region"]);
$ StrUTF8Address = GB2UTF8_FILE ($ arrAddr [$ I] ["address"]);
}
$ DblTimeDuration = GetMicroTime ()-$ dblTimeStart;
// The evaluation is complete and the result is output
Echo $ dblTimeDuration; echo "\ r \ n ";
?>
Evaluate the results twice (precise to three decimal places, in seconds ):

MySQL Query conversion: 0.112
Text query conversion: 10.590

MySQL Query conversion: 0.099
Text query conversion: 10.623

The MySQL method is far ahead of the File query method. The delimiter is a text file in the following format:

0x2121 0x3000 # IDEOGRAPHIC SPACE
0x2122 0x3001 # IDEOGRAPHIC COMMA
0x2123 0x3002 # IDEOGRAPHIC FULL STOP
0x2124 0x30FB # KATAKANA MIDDLE DOT
0x2125 0x02C9 # modifier letter macron (Mandarin Chinese first tone)
......
0x552A 0x6458 #
0x552B 0x658B #
0x552C 0x5B85 #
0x552D 0x7A84 #
......
0x777B 0x9F37 #
0x777C 0x9F3D #
0x777D 0x9F3E #
0x777E 0x9F44 #

Text files are less efficient. Therefore, you need to convert a text file to a binary file and then use the half-fold method to find the file without reading the entire file into the memory. The file format is: the file header 2 bytes, the number of records stored; then one record is saved to the file, each record 4 bytes, the first 2 bytes corresponds to the GB code, the last 2 bytes corresponds to the Unicode code. The conversion procedure is as follows:

$ ArrLines = file ("gb_unicode.txt ");
Foreach ($ arrLines as $ strLine ){
$ ArrCodeTable [hexdec (substr ($ strLine, 0, 6)] = hexdec (substr ($ strLine, 7, 6 ));
}
Ksort ($ arrCodeTable );
$ IntCount = count ($ arrCodeTable );
$ StrCount = chr ($ intCount % 256). chr (floor ($ intCount/256 ));
$ FileGBU = fopen ("gbu. dat", "wb ");
Fwrite ($ fileGBU, $ strCount );
Foreach ($ arrCodeTable as $ k => $ v ){
$ StrData = chr ($ k % 256 ). chr (floor ($ k/256 )). chr ($ v % 256 ). chr (floor ($ v/256 ));
Fwrite ($ fileGBU, $ strData );
}
Fclose ($ fileGBU );
?>
After the program is executed, the binary GB-> Unicode table gbu. dat is obtained, and the data records are sorted by the GB code for easy searching. Functions for transcoding using gbu. dat are as follows:

Function GB2UTF8_FILE1 ($ strGB ){
If (! Trim ($ strGB) return $ strGB;
$ FileGBU = fopen ("gbu. dat", "rb ");
$ StrBuf = fread ($ fileGBU, 2 );
$ IntCount = ord ($ strBuf {0}) + 256 * ord ($ strBuf {1 });
$ StrRet = "";
$ IntLen = strlen ($ strGB );
For ($ I = 0; $ I <$ intLen; $ I ++ ){
If (ord ($ strGB {$ I})> 127 ){
$ StrCurr = substr ($ strGB, $ I, 2 );
$ IntGB = hexdec (bin2hex ($ strCurr)-0x8080;
$ IntStart = 1;
$ IntEnd = $ intCount;
While ($ intStart <$ intEnd-1) {// half-way query
$ IntMid = floor ($ intStart + $ intEnd)/2 );
$ Effecffset = 2 + 4 * ($ intMid-1 );
Fseek ($ fileGBU, $ effecffset );
$ StrBuf = fread ($ fileGBU, 2 );
$ IntCode = ord ($ strBuf {0}) + 256 * ord ($ strBuf {1 });
If ($ intGB = $ intCode ){
$ IntStart = $ intMid;
Break;
}
If ($ intGB> $ intCode) $ intStart = $ intMid;
Else $ intEnd = $ intMid;
}
$ Effecffset = 2 + 4 * ($ intStart-1 );
Fseek ($ fileGBU, $ effecffset );
$ StrBuf = fread ($ fileGBU, 2 );
$ IntCode = ord ($ strBuf {0}) + 256 * ord ($ strBuf {1 });
If ($ intGB = $ intCode ){
$ StrBuf = fread ($ fileGBU, 2 );
$ IntCodeU = ord ($ strBuf {0}) + 256 * ord ($ strBuf {1 });
$ StrRet. = u2utf8 ($ intCodeU );
} Else {
$ StrRet. = "?? ";
}
$ I ++;
} Else {
$ StrRet. = $ strGB {$ I };
}
}
Return $ strRet;
}
Add it to the original evaluation program and evaluate the three methods twice at the same time to obtain the data (precise to three decimal places, in seconds ):

MySQL method: 0.125
Text file method: 10.873
Binary file half method: 0.106

MySQL method: 0.102
Text file method: 10.677
Binary file half method: 0.092

It can be seen that the binary file half method is slightly better than the MySQL method. However, all of the above tests transcode short geographic locations. what if we transcode long texts? I found the RSS 2.0 files of five blogs, all of which are GB2312 encoded. Three methods of evaluation are used to encode five files. two measurements are as follows (precise to three decimal places, in seconds ):

MySQL method: 7.206
Text file method: 0.772
Binary file half method: 5.022

MySQL method: 7.440
Text file method: 0.766
Binary file half method: 5.055

It can be seen that the optimal method is to use text files for long texts, because after the transcoding table is read into the memory, transcoding can be very efficient. In this case, we can try to improve the method of reading the text file from the binary file gbu. dat to the memory instead of the text file. The evaluation data is as follows (the accuracy and unit are the same as above ):

Read from text files: 0.766
Read the table from the binary file: 0.831

Read from text files: 0.774
Read the table from the binary file: 0.833

This improvement failed. it is more efficient to read the transcoding table from a text file.

Summary: The use of PHP to the dynamic conversion of GB encoding to UTF-8 encoding, if each conversion of the text is very small, suitable for the use of binary files combined with the half-way conversion; if each conversion of the text is large, it is suitable to use the text file storage transcoding table, and read the table into the memory at one time before conversion.