PHP file_get_contents crawl Three ways to troubleshoot gzip Web page garbled

Using the file_get_contents () function to crawl Web pages can be garbled. There are two reasons can lead to garbled, one is the encoding problem, one is the target page opened gzip, the following is open the GZIP function how to not garbled method

Turn the crawled content down to the code ($content =iconv ("GBK", "Utf-8//ignore", $content);), we're talking about crawling the gzip page. How do you judge it? The content-encoding:gzip description in the obtained header is gzip compressed. Take a look at the Firebug to know the page opened gzip not. Here is the header information for viewing my blog with Firebug, Gzip is open.
Request header information Raw header information

Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8accept-encoding gzip, Deflateaccept-language zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3connection Keep-alivecookie __utma= 225240837.787252530.1317310581.1335406161.1335411401.1537; __utmz=225240837.1326850415.887.3.utmcsr=google|utmccn= (Organic) |utmcmd=organic|utmctr=%e4%bb%bb%e4%bd%95%e9% A1%b9%e7%9b%ae%e9%83%bd%e4%b8%8d%e4%bc%9a%e9%82%a3%e4%b9%88%e7%ae%80%e5%8d%95%20site%3awww.nowamagic.net; PHPSESSID=888MJ4425P8S0M7S0FRRE3OVC7; __utmc=225240837; __utmb=225240837.1.10.1335411401host www.nowamagic.netUser-Agent mozilla/5.0 (Windows NT 5.1; rv:12.0) gecko/20100101 firefox/12.0

Here are some solutions:

1. Using your own zlib library
If the server has installed the Zlib library, use the following code can easily solve the garbled problem.

$data = file_get_contents ("compress.zlib://". $url);

2. Use Curl instead of file_get_contents

function Curl_get ($url, $gzip =false) {$curl = Curl_init ($url); curl_setopt ($curl, Curlopt_returntransfer, 1); Curl_ Setopt ($curl, Curlopt_connecttimeout, 10); if ($gzip) curl_setopt ($curl, curlopt_encoding, "gzip"); The key here $content = Curl_exec ($curl); Curl_close ($curl); return $content;}

3. Using the GZIP decompression function

function Gzdecode ($data) {$len = strlen ($data);  if ($len < | | strcmp (substr ($data, 0,2), "\x1f\x8b")) {return null;  Not GZIP format (see RFC 1952)} $method = Ord (substr ($data, 2, 1));  Compression Method $flags = Ord (substr ($data, 3, 1));   Flags if ($flags &! = $flags) {//Reserved bits is set--not allowed by RFC 1952 return null;   }//Note: $mtime may negative (PHP integer limitations) $mtime = Unpack ("V", substr ($data, bis));   $mtime = $mtime [1];   $XFL = substr ($data, 8, 1);   $os = substr ($data, 8, 1);   $headerlen = 10;   $extralen = 0;   $extra = ""; if ($flags & 4) {//2-byte length prefixed EXTRA data in header if ($len-$headerlen-2 < 8) {RET    Urn false;     Invalid format} $extralen = Unpack ("V", substr ($data, 8,2));     $extralen = $extralen [1];    if ($len-$headerlen-2-$extralen < 8) {return false; Invalid format} $extra = substr ($data, ten, $extraLen);   $headerlen + = 2 + $extralen;   } $filenamelen = 0;   $filename = ""; if ($flags & 8) {//C-style string file NAME data in header if ($len-$headerlen-1 < 8) {return    False     Invalid format} $filenamelen = Strpos (substr ($data, 8+ $extralen), Chr (0));    if ($filenamelen = = = False | | $len-$headerlen-$filenamelen-1 < 8) {return false;     Invalid format} $filename = substr ($data, $headerlen, $filenamelen);   $headerlen + = $filenamelen + 1;   } $commentlen = 0;   $comment = ""; if ($flags &) {//C-style string COMMENT data in header if ($len-$headerlen-1 < 8) {return F    Alse;     Invalid format} $commentlen = Strpos (substr ($data, 8+ $extralen + $filenamelen), Chr (0));    if ($commentlen = = = False | | $len-$headerlen-$commentlen-1 < 8) {return false;     Invalid header Format} $comment = substr ($data, $headerlen, $commentlen); $headerlen + = $commentlEn + 1;   } $HEADERCRC = "";       if ($flags & 1) {//2-bytes (lowest order) of CRC32 on header present if ($len-$headerlen-2 < 8) {    return false;     Invalid format} $CALCCRC = Crc32 (substr ($data, 0, $headerlen)) & 0xFFFF;     $HEADERCRC = Unpack ("V", substr ($data, $headerlen, 2));     $HEADERCRC = $HEADERCRC [1];    if ($HEADERCRC! = $CALCCRC) {return false;   Bad Header CRC} $headerlen + = 2;   }//GZIP Footer-these is negative due to PHP ' s limitations $DATACRC = Unpack ("V", substr ($data, -8,4));   $DATACRC = $DATACRC [1];   $isize = Unpack ("V", substr ($data,-4));  $isize = $isize [1];   Perform the decompression: $bodylen = $len-$headerlen-8;     if ($bodylen < 1) {//This should never happen-implementation bug!   return null;   } $body = substr ($data, $headerlen, $bodylen);   $data = "";     if ($bodylen > 0) {switch ($method) {case 8://Currently the only supported compression method:    $data = Gzinflate ($body);       Break     Default://Unknown compression method return false;     }} else {//I ' m not sure if zero-byte body content is allowed.  Allow it for now ...  Do nothing ...} verifiy decompressed size and CRC32://Note:this may fail with large data sizes depending on how//PHP ' s I   Nteger limitations affect strlen () since $isize//May is negative for large sizes.  if ($isize! = strlen ($data) | | CRC32 ($data)! = $DATACRC) {//Bad format!     Length or CRC doesn ' t match!   return false; } return $data; }

Use:

$html =file_get_contents (' http://www.jb51.net/'); $html =gzdecode ($html);

The introduction of these three methods, should be able to solve the majority of gzip caused by the crawl garbled problem.

