PHP Multi-threaded crawl Web page code sharing

This article introduces, with PHP implementation of multi-threaded crawl Web page code, there is a need for friends reference. In PHP, you can use curl to perform various transfers of files, such as simulating a browser sending a get,post request. The PHP language itself does not support multi-threading, so the development of the crawler is not efficient, so the use of Curl Multi Functions to achieve concurrent multi-threaded access to multiple URL addresses. For the basic content of curl, you can refer to the following article: PHP Curl application Example Analysis PHP Curl Usage Example code PHP Curl Learning Summary This section describes an example of using Curl Multi Functions to download files in parallel with multithreading. Example 1, get content to write directly to a file !--? php/*** multi-threaded Crawl Web content * Edit by bbs.it-home.org*/$urls = Array (' HTTP://WW w.sina.com.cn/', ' http://www.sohu.com/', ' http://bbs.it-home.org/'); The URL of the page to crawl $save _to= '/test.txt '; Write file to crawl content $st = fopen ($save _to, "a"); $MH = Curl_multi_init (); foreach ($urls as $i =--> $url) {$conn [$i] = Curl_init ($url); curl_setopt ($conn [$i], Curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) "); curl_setopt ($conn [$i], Curlopt_header, 0); curl_setopt ($conn [$i], curlopt_connecttimeout,60); curl_setopt ($conn [$i], Curlopt_file, $st); Writes the crawled code to the file Curl_multi_add_handle ($MH, $conn [$i]); }//Initialize do {curl_multi_exec ($MH, $active); } while ($active); Executes a foreach ($urls as $i = = $url) {curl_multi_remove_handle ($MH, $conn [$i]); Curl_close ($conn [$i]); }//End cleanup Curl_multi_close ($MH); Fclose ($st);; Example 2, get content into a variable, and then write to a file $url) {$conn [$i] = Curl_init ($url); curl_setopt ($conn [$i], Curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) "); curl_setopt ($conn [$i], Curlopt_header, 0); curl_setopt ($conn [$i], curlopt_connecttimeout,60); curl_setopt ($conn [$i],curlopt_returntransfer,true); Set the crawl substitution code into a string, not output to the browser curl_multi_add_handle ($MH, $conn [$i]); } do {curl_multi_exec ($MH, $active); } while ($active); foreach ($urls as $i + $url) {$data = Curl_multi_getcontent ($conn [$i]);//Get Crawled Code string fwrite ($st, $data); Writes a string to the file. It is also possible to deposit the database. }//Get data variable and write to file foreach ($urls as $i + = $url) {curl_multi_remove_handle ($MH, $conn [$i]); Curl_close ($conn [$i]); } curl_multi_close ($MH); Fclose ($st); ?>

