File_get_contents Crawl Web page garbled solution

Sometimes using the file_get_contents () function to crawl the Web page will be garbled phenomenon. There are two reasons to cause garbled, one is the encoding problem, one is the target page opened gzip.

Coding problem is good to do, the content of the crawl to the code can be ($content =iconv ("GBK", "Utf-8//ignore", $content);), we are here to discuss how to crawl open the Gzip page. How do you judge it? The content-encoding:gzip description in the obtained header is gzip compressed. Take a look at the Firebug to know the page opened gzip not. Here is the header information for viewing my blog with Firebug, Gzip is open. New York City Casino

Request header Information Raw header information Accepttext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8accept-encodinggzip, Deflateaccept-languagezh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3connectionkeep-alivecookie__utma= 225240837.787252530.1317310581.1335406161.1335411401.1537; __utmz=225240837.1326850415.887.3.utmcsr=google|utmccn= (Organic) |utmcmd=organic|utmctr=%e4%bb%bb%e4%bd%95%e9% A1%b9%e7%9b%ae%e9%83%bd%e4%b8%8d%e4%bc%9a%e9%82%a3%e4%b9%88%e7%ae%80%e5%8d%95%20site%3awww.nowamagic.net; PHPSESSID=888MJ4425P8S0M7S0FRRE3OVC7; __utmc=225240837; __utmb=225240837.1.10.1335411401hostwww.nowamagic.netuser-agentmozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 firefox/12.0

Here are some solutions:

1. Using your own zlib library

If the server has installed the Zlib library, use the following code can easily solve the garbled problem.

2. Use Curl instead of file_get_contents

function Curl_get ($url, $gzip =false) {$curl = Curl_init ($url); curl_setopt ($curl, Curlopt_returntransfer, 1); Curl_ Setopt ($curl, Curlopt_connecttimeout), if ($gzip) curl_setopt ($curl, curlopt_encoding, "gzip"); The key here $content = Curl_exec ($curl); Curl_close ($curl); return $content;}

3. Using the GZIP decompression function

function Gzdecode ($data) {$len = strlen ($data);  if ($len < | | strcmp (substr ($data, 0,2), "\x1f\x8b")) {return null;  Not GZIP format (see RFC 1952)} $method = Ord (substr ($data, 2, 1));  Compression Method $flags = Ord (substr ($data, 3, 1));   Flags if ($flags &! = $flags) {//Reserved bits is set--not allowed by RFC 1952 return null;   }//Note: $mtime may negative (PHP integer limitations) $mtime = Unpack ("V", substr ($data, bis));   $mtime = $mtime [1];   $XFL = substr ($data, 8, 1);   $os = substr ($data, 8, 1);   $headerlen = 10;   $extralen = 0;   $extra = ""; if ($flags & 4) {//2-byte length prefixed EXTRA data in header if ($len-$headerlen-2 < 8) {RET    Urn false;     Invalid format} $extralen = Unpack ("V", substr ($data, 8,2));     $extralen = $extralen [1];    if ($len-$headerlen-2-$extralen < 8) {return false; Invalid format} $extra = substr ($data, ten, $extraLen);   $headerlen + = 2 + $extralen;   } $filenamelen = 0;   $filename = ""; if ($flags & 8) {//C-style string file NAME data in header if ($len-$headerlen-1 < 8) {return    False     Invalid format} $filenamelen = Strpos (substr ($data, 8+ $extralen), Chr (0));    if ($filenamelen = = = False | | $len-$headerlen-$filenamelen-1 < 8) {return false;     Invalid format} $filename = substr ($data, $headerlen, $filenamelen);   $headerlen + = $filenamelen + 1;   } $commentlen = 0;   $comment = ""; if ($flags &) {//C-style string COMMENT data in header if ($len-$headerlen-1 < 8) {return F    Alse;     Invalid format} $commentlen = Strpos (substr ($data, 8+ $extralen + $filenamelen), Chr (0));    if ($commentlen = = = False | | $len-$headerlen-$commentlen-1 < 8) {return false;     Invalid header Format} $comment = substr ($data, $headerlen, $commentlen); $headerlen + = $commenTlen + 1;   } $HEADERCRC = "";       if ($flags & 1) {//2-bytes (lowest order) of CRC32 on header present if ($len-$headerlen-2 < 8) {    return false;     Invalid format} $CALCCRC = Crc32 (substr ($data, 0, $headerlen)) & 0xFFFF;     $HEADERCRC = Unpack ("V", substr ($data, $headerlen, 2));     $HEADERCRC = $HEADERCRC [1];    if ($HEADERCRC! = $CALCCRC) {return false;   Bad Header CRC} $headerlen + = 2;   }//GZIP Footer-these is negative due to PHP ' s limitations $DATACRC = Unpack ("V", substr ($data, -8,4));   $DATACRC = $DATACRC [1];   $isize = Unpack ("V", substr ($data,-4));   $isize = $isize [1];   Perform the decompression: $bodylen = $len-$headerlen-8;     if ($bodylen < 1) {//This should never happen-implementation bug!   return null;   } $body = substr ($data, $headerlen, $bodylen);   $data = ""; if ($bodylen > 0) {switch ($method) {case 8://Currently the only supported compression method:         $data = Gzinflate ($body);       Break     Default://Unknown compression method return false;     }} else {//I ' m not sure if zero-byte body content is allowed.  Allow it for now ...   Do nothing ...} verifiy decompressed size and CRC32://Note:this may fail with large data sizes depending on how//PHP ' s I   Nteger limitations affect strlen () since $isize//May is negative for large sizes.  if ($isize! = strlen ($data) | | CRC32 ($data)! = $DATACRC) {//Bad format!     Length or CRC doesn ' t match!   return false; } return $data; }

Use:

$html =file_get_contents (' http://www.nowamagic.net/librarys/veda/'); $html =gzdecode ($html);

The introduction of these three methods, should be able to solve the majority of gzip caused by the crawl garbled problem.

File_get_contents Crawl Web page garbled solution