File_get_contents captures garbled web pages. Sometimes, garbled characters occur when you use the file_get_contents () function to Capture webpages. There are two possible causes for garbled characters: encoding and Gzip on the target page. The encoding problem is good. sometimes, garbled characters occur when the file_get_contents () function is used to capture webpages. There are two possible causes for garbled characters: encoding and Gzip on the target page.
Encoding problems are easy to solve, just encode the captured content ($ content = iconv ("GBK", "UTF-8 // IGNORE", $ content );), here we will discuss how to capture the Gzip page. How to judge? Content-Encoding: gzip indicates that the Content is compressed by GZIP. If you use FireBug, you can see that gzip is not enabled on the page. The following shows the header information of my blog with firebug. Gzip is enabled.
Original request header information: Accepttext/html, application/xhtml + xml, application/xml; q = 0.9, */*; q = 0.8Accept-Encodinggzip, deflateAccept-Languagezh-cn, zh; q = 0.8, en-us; q = 0.5, en; q = 0.3Connectionkeep-aliveCookie _ utma = encrypt; _ utmz = 225240837.1326850415.887.3.utmcsr = google | utmccn =) | utmcmd = organic | utmctr = % E4 % BB % E4 % BD % 95% E9 % A1 % B9 % E7 % 9B % AE % E9 % 83% BD % E4 % B8 % 8D % E4 % BC % 9A % E9 % 82% A3 % E4 % B9 % 88% E7 % AE % 80% E5 % 8D % 95% 20 site % 3Awww.bkjia.com; PHPSESSID = 888mj4jwp8s0m7s0frre3ovc7; _ utmc = 225240837; _ utmb = runtime-AgentMozilla/5.0 (Windows NT 5.1; rv: 12.0) Gecko/20100101 Firefox/12.0
The following describes some solutions:
1. Use the built-in zlib library
If the zlib library has been installed on the server, the following code can easily solve the garbled problem.
$data = file_get_contents("compress.zlib://".$url);
2. use CURL to replace file_get_contents
Function curl_get ($ url, $ gzip = false) {$ curl = curl_init ($ url); curl_setopt ($ curl, CURLOPT_RETURNTRANSFER, 1); curl_setopt ($ curl, CURLOPT_CONNECTTIMEOUT, 10); if ($ gzip) curl_setopt ($ curl, CURLOPT_ENCODING, "gzip"); // The key here is $ content = curl_exec ($ curl); curl_close ($ curl ); return $ content ;}
3. use the gzip decompression function
function gzdecode($data) { $len = strlen($data); if ($len < 18 || strcmp(substr($data,0,2),"\x1f\x8b")) { return null; // Not GZIP format (See RFC 1952) } $method = ord(substr($data,2,1)); // Compression method $flags = ord(substr($data,3,1)); // Flags if ($flags & 31 != $flags) { // Reserved bits are set -- NOT ALLOWED by RFC 1952 return null; } // NOTE: $mtime may be negative (PHP integer limitations) $mtime = unpack("V", substr($data,4,4)); $mtime = $mtime[1]; $xfl = substr($data,8,1); $os = substr($data,8,1); $headerlen = 10; $extralen = 0; $extra = ""; if ($flags & 4) { // 2-byte length prefixed EXTRA data in header if ($len - $headerlen - 2 < 8) { return false; // Invalid format } $extralen = unpack("v",substr($data,8,2)); $extralen = $extralen[1]; if ($len - $headerlen - 2 - $extralen < 8) { return false; // Invalid format } $extra = substr($data,10,$extralen); $headerlen += 2 + $extralen; } $filenamelen = 0; $filename = ""; if ($flags & 8) { // C-style string file NAME data in header if ($len - $headerlen - 1 < 8) { return false; // Invalid format } $filenamelen = strpos(substr($data,8+$extralen),chr(0)); if ($filenamelen === false || $len - $headerlen - $filenamelen - 1 < 8) { return false; // Invalid format } $filename = substr($data,$headerlen,$filenamelen); $headerlen += $filenamelen + 1; } $commentlen = 0; $comment = ""; if ($flags & 16) { // C-style string COMMENT data in header if ($len - $headerlen - 1 < 8) { return false; // Invalid format } $commentlen = strpos(substr($data,8+$extralen+$filenamelen),chr(0)); if ($commentlen === false || $len - $headerlen - $commentlen - 1 < 8) { return false; // Invalid header format } $comment = substr($data,$headerlen,$commentlen); $headerlen += $commentlen + 1; } $headercrc = ""; if ($flags & 1) { // 2-bytes (lowest order) of CRC32 on header present if ($len - $headerlen - 2 < 8) { return false; // Invalid format } $calccrc = crc32(substr($data,0,$headerlen)) & 0xffff; $headercrc = unpack("v", substr($data,$headerlen,2)); $headercrc = $headercrc[1]; if ($headercrc != $calccrc) { return false; // Bad header CRC } $headerlen += 2; } // GZIP FOOTER - These be negative due to PHP's limitations $datacrc = unpack("V",substr($data,-8,4)); $datacrc = $datacrc[1]; $isize = unpack("V",substr($data,-4)); $isize = $isize[1]; // Perform the decompression: $bodylen = $len-$headerlen-8; if ($bodylen < 1) { // This should never happen - IMPLEMENTATION BUG! return null; } $body = substr($data,$headerlen,$bodylen); $data = ""; if ($bodylen > 0) { switch ($method) { case 8: // Currently the only supported compression method: $data = gzinflate($body); break; default: // Unknown compression method return false; } } else { // I'm not sure if zero-byte body content is allowed. // Allow it for now... Do nothing... } // Verifiy decompressed size and CRC32: // NOTE: This may fail with large data sizes depending on how // PHP's integer limitations affect strlen() since $isize // may be negative for large sizes. if ($isize != strlen($data) || crc32($data) != $datacrc) { // Bad format! Length or CRC doesn't match! return false; } return $data; }
Usage:
$html=file_get_contents('http://www.bkjia.com/librarys/veda/');$html=gzdecode($html);
This article introduces these three methods to solve the garbled crawling problem caused by most gzip files.
The file_get_contents () function captures webpages with garbled characters. There are two possible causes for garbled characters: encoding and Gzip on the target page. Encoding problems...