Three solutions for phpfile_get_contents crawling garbled Gzip web pages

Garbled characters may occur when a webpage is crawled using the file_get_contents () function. There are two possible causes for garbled characters: encoding and Gzip on the target page, the following is how to enable the Gzip function can not garbled method to capture the content of the transfer to the code ($ content = iconv ("GBK", "UTF-8 // IGNORE ", $ content);). here we will discuss how to capture the Gzip page. How to judge? Content-Encoding: gzip indicates that the Content is compressed by GZIP. If you use FireBug, you can see that gzip is not enabled on the page. The following shows the header information of my blog with firebug. Gzip is enabled.
The code is as follows:
Original request header information
Accept text/html, application/xhtml + xml, application/xml; q = 0.9, */*; q = 0.8
Accept-Encoding gzip, deflate
Accept-Language zh-cn, zh; q = 0.8, en-us; q = 0.5, en; q = 0.3
Connection keep-alive
Cookie _ utma = canonical; _ utmz = 225240837.1326850415.887.3.utmcsr = google | utmccn = (organic) | utmcmd = organic | utmctr = % E4 % BB % E4 % BD % 95% E9 % A1 % B9 % E7 % 9B % AE % E9 % 83% BD % E4 % B8 % 8D % E4 % BC % 9A % E9 % 82% A3 % E4 % B9 % 88% E7 % AE % 80% E5 % 8D % 95% 20 site % 3Awww.nowamagic.net; PHPSESSID = 888mj4jwp8s0m7s0frre3ovc7; _ utmc = 225240837; _ utmb = 225240837.1.10.1335411401
Host www.nowamagic.net
User-Agent Mozilla/5.0 (Windows NT 5.1; rv: 12.0) Gecko/20100101 Firefox/12.0

The following describes some solutions:

1. Use the built-in zlib library
If the zlib library has been installed on the server, the following code can easily solve the garbled problem.
The code is as follows:
$ Data = file_get_contents ("compress. zlib: //". $ url );

2. use CURL to replace file_get_contents
The code is as follows:
Function curl_get ($ url, $ gzip = false ){
$ Curl = curl_init ($ url );
Curl_setopt ($ curl, CURLOPT_RETURNTRANSFER, 1 );
Curl_setopt ($ curl, CURLOPT_CONNECTTIMEOUT, 10 );
If ($ gzip) curl_setopt ($ curl, CURLOPT_ENCODING, "gzip"); // The key here
$ Content = curl_exec ($ curl );
Curl_close ($ curl );
Return $ content;
}

3. use the gzip decompression function
The code is as follows:
Function gzdecode ($ data ){
$ Len = strlen ($ data );
If ($ len <18 | strcmp (substr ($ data, 0, 2), "\ x1f \ x8b ")){
Return null; // Not GZIP format (See RFC 1952)
}
$ Method = ord (substr ($ data, 2, 1); // Compression method
$ Flags = ord (substr ($ data, 3, 1); // Flags
If ($ flags & 31! = $ Flags ){
// Reserved bits are set -- not allowed by RFC 1952.
Return null;
}
// NOTE: $ mtime may be negative (PHP integer limitations)
$ Mtime = unpack ("V", substr ($ data, 4, 4 ));
$ Mtime = $ mtime [1];
$ Xfl = substr ($ data, 8, 1 );
$ OS = substr ($ data, 8, 1 );
$ Headerlen = 10;
$ Extralen = 0;
$ Extra = "";
If ($ flags & 4 ){
// 2-byte length prefixed EXTRA data in header
If ($ len-$ headerlen-2 <8 ){
Return false; // Invalid format
}
$ Extralen = unpack ("v", substr ($ data, 8, 2 ));
$ Extralen = $ extralen [1];
If ($ len-$ headerlen-2-$ extralen <8 ){
Return false; // Invalid format
}
$ Extra = substr ($ data, 10, $ extralen );
$ Headerlen + = 2 + $ extralen;
}

$ Filenamelen = 0;
$ Filename = "";
If ($ flags & 8 ){
// C-style string file NAME data in header
If ($ len-$ headerlen-1 <8 ){
Return false; // Invalid format
}
$ Filenamelen = strpos (substr ($ data, 8 + $ extralen), chr (0 ));
If ($ filenamelen === false | $ len-$ headerlen-$ filenamelen-1 <8 ){
Return false; // Invalid format
}
$ Filename = substr ($ data, $ headerlen, $ filenamelen );
$ Headerlen + = $ filenamelen + 1;
}

$ Commentlen = 0;
$ Comment = "";
If ($ flags & 16 ){
// C-style string COMMENT data in header
If ($ len-$ headerlen-1 <8 ){
Return false; // Invalid format
}
$ Commentlen = strpos (substr ($ data, 8 + $ extralen + $ filenamelen), chr (0 ));
If ($ commentlen = false | $ len-$ headerlen-$ commentlen-1 <8 ){
Return false; // Invalid header format
}
$ Comment = substr ($ data, $ headerlen, $ commentlen );
$ Headerlen + = $ commentlen + 1;
}

$ Headercrc = "";
If ($ flags & 1 ){
// 2-bytes (lowest order) of CRC32 on header present
If ($ len-$ headerlen-2 <8 ){
Return false; // Invalid format
}
$ Calccrc = crc32 (substr ($ data, 0, $ headerlen) & 0 xffff;
$ Headercrc = unpack ("v", substr ($ data, $ headerlen, 2 ));
$ Headercrc = $ headercrc [1];
If ($ headercrc! = $ Calccrc ){
Return false; // Bad header CRC
}
$ Headerlen + = 2;
}

// Gzip footer-These be negative due to PHP's limitations
$ Datacrc = unpack ("V", substr ($ data,-8, 4 ));
$ Datacrc = $ datacrc [1];
$ Isize = unpack ("V", substr ($ data,-4 ));
$ Isize = $ isize [1];

// Perform the decompression:
$ Bodylen = $ len-$ headerlen-8;
If ($ bodylen <1 ){
// This shoshould never happen-implementation bug!
Return null;
}
$ Body = substr ($ data, $ headerlen, $ bodylen );
$ Data = "";
If ($ bodylen> 0 ){
Switch ($ method ){
Case 8:
// Currently the only supported compression method:
$ Data = gzinflate ($ body );
Break;
Default:
// Unknown compression method
Return false;
}
} Else {
// I'm not sure if zero-byte body content is allowed.
// Allow it for now... Do nothing...
}

// Verifiy decompressed size and CRC32:
// NOTE: This may fail with large data sizes depending on how
// PHP's integer limitations affect strlen () since $ isize
// May be negative for large sizes.
If ($ isize! = Strlen ($ data) | crc32 ($ data )! = $ Datacrc ){
// Bad format! Length or CRC doesn' t match!
Return false;
}
Return $ data;
}

Usage:
The code is as follows:
$ Html = file_get_contents ('http: // www.bitsCN.com /');
$ Html = gzdecode ($ html );

This article introduces these three methods to solve the garbled crawling problem caused by most gzip files.