PHP Multi-threaded crawl Web implementation Code _php tutorial

Due to the PHP language itself does not support multi-threading, so the development of crawler efficiency is not high, this time often need to use Curl Multi Functions It can achieve concurrent multi-threaded access to multiple URL addresses. Since Curl Multi function is so powerful, can you use Curl Multi Functions to write concurrent multi-threaded download files, of course, the following gives my code:

Code Listing 1: Writing the obtained code directly to a file
Copy CodeThe code is as follows:
$urls = Array (
' http://www.sina.com.cn/',
' http://www.sohu.com/',
' http://www.163.com/'
); Set the URL of the page to crawl

$save _to= '/test.txt '; Write the crawled code to the file

$st = fopen ($save _to, "a");
$MH = Curl_multi_init ();

foreach ($urls as $i = = $url) {
$conn [$i] = Curl_init ($url);
curl_setopt ($conn [$i], Curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) ");
curl_setopt ($conn [$i], Curlopt_header, 0);
curl_setopt ($conn [$i], curlopt_connecttimeout,60);
curl_setopt ($conn [$i], Curlopt_file, $st); Set the crawled code to write to the file
Curl_multi_add_handle ($MH, $conn [$i]);
}//Initialization

do {
Curl_multi_exec ($MH, $active);
} while ($active); Perform

foreach ($urls as $i = = $url) {
Curl_multi_remove_handle ($MH, $conn [$i]);
Curl_close ($conn [$i]);
}//End cleanup

Curl_multi_close ($MH);
Fclose ($st);
?>

Code Listing 2: Putting the obtained code into a variable before writing to a file
Copy CodeThe code is as follows:
$urls = Array (
' http://www.sina.com.cn/',
' http://www.sohu.com/',
' http://www.163.com/'
);

$save _to= '/test.txt '; Write the crawled code to the file
$st = fopen ($save _to, "a");

$MH = Curl_multi_init ();
foreach ($urls as $i = = $url) {
$conn [$i] = Curl_init ($url);
curl_setopt ($conn [$i], Curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) ");
curl_setopt ($conn [$i], Curlopt_header, 0);
curl_setopt ($conn [$i], curlopt_connecttimeout,60);
curl_setopt ($conn [$i],curlopt_returntransfer,true); Setting does not write the crawl substitution code to the browser, but instead to the string
Curl_multi_add_handle ($MH, $conn [$i]);
}

do {
Curl_multi_exec ($MH, $active);
} while ($active);

foreach ($urls as $i = = $url) {
$data = Curl_multi_getcontent ($conn [$i]); Gets the code string for the crawl
Fwrite ($st, $data); Writes a string to the file. Of course, you can also not write to the file, such as deposit database
}//Get data variable and write to file

foreach ($urls as $i = = $url) {
Curl_multi_remove_handle ($MH, $conn [$i]);
Curl_close ($conn [$i]);
}

Curl_multi_close ($MH);
Fclose ($st);
?>

http://www.bkjia.com/PHPjc/322228.html www.bkjia.com true http://www.bkjia.com/PHPjc/322228.html techarticle Limited by the PHP language itself does not support multi-threading, so the development of crawler efficiency is not high, this time often need to rely on Curl Multi Functions It can achieve concurrent multi-threaded access to multiple ...

