PHP uses Curl function to crawl Web pages and download files in multiple threads

PHP uses Curl Functions to complete a variety of transfer file operations, such as simulating the browser to send Get,post requests, and so on, but because the PHP language itself does not support multi-threading, so the development of the crawler is not very efficient, so often need to use Curl Multi Functions This feature to achieve concurrent multi-threaded access to multiple URL addresses to implement concurrent multi-threaded crawl Web pages or download files, as for the specific implementation process, please refer to the following examples:

(1) The following code is implemented to crawl multiple URLs, and then the crawled URL of the page code to write to the specified file

Fclose ($st);

(2) The code below is almost the same as above, except that this place is where the obtained code is first put into the variable and then the obtained content is written to the specified file

$urls = Array (
' http://www.scutephp.com/',
' http://www.google.com/',
' http://www.example.com/'
);
$save _to= '/test.txt '; Write the crawled code to the file
$st = fopen ($save _to, "a");
$MH = Curl_multi_init ();
foreach ($urls as $i = = $url) {
$conn [$i] = Curl_init ($url);
curl_setopt ($conn [$i], Curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) ");
curl_setopt ($conn [$i], Curlopt_header, 0);
curl_setopt ($conn [$i], curlopt_connecttimeout,60);
curl_setopt ($conn [$i],curlopt_returntransfer,true); Do not write crawl substitution codes to the browser, but instead convert them to strings
Curl_multi_add_handle ($MH, $conn [$i]);
}
do {
Curl_multi_exec ($MH, $active);
} while ($active);
foreach ($urls as $i = = $url) {
$data = Curl_multi_getcontent ($conn [$i]); Gets the code string for the crawl
Fwrite ($st, $data); Writing a string to a file
}//Get data variable and write to file
foreach ($urls as $i = = $url) {
Curl_multi_remove_handle ($MH, $conn [$i]);
Curl_close ($conn [$i]);
}
Curl_multi_close ($MH);
Fclose ($st);

(3) The following code is implemented using the PHP Curl Functions implementation of concurrent multi-threaded download files

$urls =array (
' Http://www.scutephp.com/5w.zip ',
' Http://www.scutephp.com/5w.zip ',
' Http://www.scutephp.com/5w.zip '
);
$save _to= './home/';
$MH =curl_multi_init ();
foreach ($urls as $i = = $url) {
$g = $save _to.basename ($url);
if (!is_file ($g)) {
$conn [$i]=curl_init ($url);
$fp [$i]=fopen ($g, "w");
curl_setopt ($conn [$i],curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) ");
curl_setopt ($conn [$i],curlopt_file, $fp [$i]);
curl_setopt ($conn [$i],curlopt_header, 0);
curl_setopt ($conn [$i],curlopt_connecttimeout,60);
Curl_multi_add_handle ($MH, $conn [$i]);
}
}
do{
$n =curl_multi_exec ($MH, $active);
}while ($active);
foreach ($urls as $i = = $url) {
Curl_multi_remove_handle ($MH, $conn [$i]);
Curl_close ($conn [$i]);
Fclose ($fp [$i]);
}
Curl_multi_close ($MH); $urls =array (
' Http://www.scutephp.com/5w.zip ',
' Http://www.scutephp.com/5w.zip ',
' Http://www.scutephp.com/5w.zip '
);
$save _to= './home/';
$MH =curl_multi_init ();
foreach ($urls as $i = = $url) {
$g = $save _to.basename ($url);
if (!is_file ($g)) {
$conn [$i]=curl_init ($url);
$fp [$i]=fopen ($g, "w");
curl_setopt ($conn [$i],curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) ");
curl_setopt ($conn [$i],curlopt_file, $fp [$i]);
curl_setopt ($conn [$i],curlopt_header, 0);
curl_setopt ($conn [$i],curlopt_connecttimeout,60);
Curl_multi_add_handle ($MH, $conn [$i]);
}
}
do{
$n =curl_multi_exec ($MH, $active);
}while ($active);
foreach ($urls as $i = = $url) {
Curl_multi_remove_handle ($MH, $conn [$i]);
Curl_close ($conn [$i]);
Fclose ($fp [$i]);
}
Curl_multi_close ($MH);

