PHP file_get_contents Crawl gzip page garbled three ways to solve _php tutorial

Turn the crawled content down to the code ($content =iconv ("GBK", "Utf-8//ignore", $content);), we're talking about crawling the gzip page. How do you judge it? The content-encoding:gzip description in the obtained header is gzip compressed. Take a look at the Firebug to know the page opened gzip not. Here is the header information for viewing my blog with Firebug, Gzip is open.
Copy CodeThe code is as follows:
Request header information Raw header information
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-encoding gzip, deflate
Accept-language zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3
Connection keep-alive
Cookie __utma=225240837.787252530.1317310581.1335406161.1335411401.1537; __utmz=225240837.1326850415.887.3.utmcsr=google|utmccn= (Organic) |utmcmd=organic|utmctr=%e4%bb%bb%e4%bd%95%e9% A1%b9%e7%9b%ae%e9%83%bd%e4%b8%8d%e4%bc%9a%e9%82%a3%e4%b9%88%e7%ae%80%e5%8d%95%20site%3awww.nowamagic.net; PHPSESSID=888MJ4425P8S0M7S0FRRE3OVC7; __utmc=225240837; __utmb=225240837.1.10.1335411401
Host www.nowamagic.net
User-agent mozilla/5.0 (Windows NT 5.1; rv:12.0) gecko/20100101 firefox/12.0

Here are some solutions:

1. Using your own zlib library
If the server has installed the Zlib library, use the following code can easily solve the garbled problem.
Copy the Code code as follows:
$data = file_get_contents ("compress.zlib://". $url);

2. Use Curl instead of file_get_contents
Copy the Code code as follows:
function Curl_get ($url, $gzip =false) {
$curl = Curl_init ($url);
curl_setopt ($curl, Curlopt_returntransfer, 1);
curl_setopt ($curl, Curlopt_connecttimeout, 10);
if ($gzip) curl_setopt ($curl, curlopt_encoding, "gzip"); Here's the point.
$content = curl_exec ($curl);
Curl_close ($curl);
return $content;
}

3. Using the GZIP decompression function
Copy CodeThe code is as follows:
function Gzdecode ($data) {
$len = strlen ($data);
if ($len < | | strcmp (substr ($data, 0,2), "\x1f\x8b")) {
return null; Not GZIP format (see RFC 1952)
}
$method = Ord (substr ($data, 2, 1)); Compression method
$flags = Ord (substr ($data, 3, 1)); Flags
if ($flags &! = $flags) {
Reserved bits is set--not allowed by RFC 1952
return null;
}
Note: $mtime may negative (PHP integer limitations)
$mtime = Unpack ("V", substr ($data, bis));
$mtime = $mtime [1];
$XFL = substr ($data, 8, 1);
$os = substr ($data, 8, 1);
$headerlen = 10;
$extralen = 0;
$extra = "";
if ($flags & 4) {
2-byte length prefixed EXTRA data in header
if ($len-$headerlen-2 < 8) {
return false; Invalid format
}
$extralen = Unpack ("V", substr ($data, 8,2));
$extralen = $extralen [1];
if ($len-$headerlen-2-$extralen < 8) {
return false; Invalid format
}
$extra = substr ($data, ten, $extralen);
$headerlen + = 2 + $extralen;
}

$filenamelen = 0;
$filename = "";
if ($flags & 8) {
//C-style string file NAME data in header
if ($len-$headerlen-1 < 8) {
R Eturn false; Invalid format
}
$filenamelen = Strpos (substr ($data, 8+ $extralen), Chr (0));
if ($filenamelen = = = False | | $len-$headerlen-$filenamelen-1 < 8) {
return false;//Invalid format
}
$filename = substr ($data, $headerlen, $filenamelen);
$headerlen + = $filenamelen + 1;
}

$commentlen = 0;
$comment = "";
if ($flags & 16) {
C-style string COMMENT data in header
if ($len-$headerlen-1 < 8) {
return false; Invalid format
}
$commentlen = Strpos (substr ($data, 8+ $extralen + $filenamelen), Chr (0));
if ($commentlen = = = False | | $len-$headerlen-$commentlen-1 < 8) {
return false; Invalid Header Format
}
$comment = substr ($data, $headerlen, $commentlen);
$headerlen + = $commentlen + 1;
}

$HEADERCRC = "";
if ($flags & 1) {
2-bytes (lowest order) of CRC32 on header present
if ($len-$headerlen-2 < 8) {
return false; Invalid format
}
$CALCCRC = CRC32 (substr ($data, 0, $headerlen)) & 0xFFFF;
$HEADERCRC = Unpack ("V", substr ($data, $headerlen, 2));
$HEADERCRC = $HEADERCRC [1];
if ($HEADERCRC! = $CALCCRC) {
return false; Bad Header CRC
}
$headerlen + = 2;
}

GZIP Footer-these be negative due to PHP limitations
$DATACRC = Unpack ("V", substr ($data, -8,4));
$DATACRC = $DATACRC [1];
$isize = Unpack ("V", substr ($data,-4));
$isize = $isize [1];

Perform the Decompression:
$bodylen = $len-$headerlen-8;
if ($bodylen < 1) {
This should never happen-implementation bug!
return null;
}
$body = substr ($data, $headerlen, $bodylen);
$data = "";
if ($bodylen > 0) {
Switch ($method) {
Case 8:
Currently the only supported compression method:
$data = Gzinflate ($body);
Break
Default
Unknown compression method
return false;
}
} else {
I ' m not sure if zero-byte body content is allowed.
Allow it for now ... Do nothing ...
}

verifiy decompressed size and CRC32:
Note:this may fail with large data sizes depending on how
PHP ' s integer limitations affect strlen () since $isize
May is negative for large sizes.
if ($isize! = strlen ($data) | | CRC32 ($data)! = $DATACRC) {
Bad format! Length or CRC doesn ' t match!
return false;
}
return $data;
}

Use:
Copy the Code code as follows:
$html =file_get_contents (' http://www.jb51.net/');
$html =gzdecode ($html);

The introduction of these three methods, should be able to solve the majority of gzip caused by the crawl garbled problem.

http://www.bkjia.com/PHPjc/825112.html www.bkjia.com true http://www.bkjia.com/PHPjc/825112.html techarticle Turn the crawled content down to the code ($content =iconv ("GBK", "Utf-8//ignore", $content);), we're talking about crawling the gzip page. How do you judge it? Get the head ...

