PHP with Curl for multi-threaded crawl, Phpcurl multi-threaded crawl _php tutorial

PHP with Curl for multi-threaded crawl, Phpcurl multi-threaded crawl

PHP combined with curl for multi-threaded crawling

<?php/*curl Multi-threaded CRAWL *//** * Curl Multithreading * * @param array $array parallel URL * @param int $timeout Time Out * @return Arra Y */function Curl_http ($array, $timeout) {$res = array (); $mh = Curl_multi_init ();//Create multiple Curl handles $startime = Getmicrotim E ();    foreach ($array as $k = + $url) {$conn [$k]=curl_init ($url); curl_setopt ($conn [$k], curlopt_timeout, $timeout);//Set time-out curl_setopt ($conn [$k], Curlopt_useragent, ' mozilla/5.0 ( Compatible MSIE 5.01;    Windows NT 5.0) ');    curl_setopt ($conn [$k], Curlopt_maxredirs, 7),//http orientation level curl_setopt ($conn [$k], Curlopt_header, 0);//Do not HEADER, add block efficiency curl_setopt ($conn [$k], curlopt_followlocation, 1);    302 redirect Curl_setopt ($conn [$k],curlopt_returntransfer,1); Curl_multi_add_handle ($MH, $conn [$k]); }//Prevent death cycle consumes the CPU this paragraph is based on the online notation do {$MRC = Curl_multi_exec ($MH, $active);//When no data, active=true} while ($MRC = = Curlm_call_multi_p Erform);//When data is being accepted while ($active and $MRC = = CURLM_OK) {//when there is no data or when a request is paused, active=true if (Curl_multi_select ($MH)! =-1) { do {$MRC = Curl_multi_exec ($MH, $active);  } while ($MRC = = Curlm_call_multi_perform);    }} foreach ($array as $k = + $url) {curl_error ($conn [$k]); $res [$k]=curl_multi_getcontent ($conn [$k]);//obtain return information $header [$k]=curl_getinfo ($conn [$k]);//Return header information Curl_close ($conn [  $K]);//close handle Curl_multi_remove_handle ($MH, $conn [$k]); Free resources} curl_multi_close ($MH); $endtime = Getmicrotime (); $diff _time = $endtime-$startime; Return Array (' diff_time ' = = $diff _time, ' return ' and ' $res, ' header ' = ' $header ');   }//Calculates the current time function getmicrotime () {list ($usec, $sec) = Explode ("", Microtime ()); return (float) $usec + (float) $sec); }//test, curl three urls $array = Array ("http://www.weibo.com/", "http://www.renren.com/", "http://www.qq.com/"); $data = Curl_http ($array, ' 10 ');//Call Var_dump ($DATA);//output//If the post data is greater than 1024 bytes, Curl does not directly initiate a POST request//Send request. The header contains an empty expect. curl_setopt ($ch, Curlopt_httpheader, Array ("Expect:"));? >

Let's look at a few more examples.

(1) The following code is implemented to crawl multiple URLs, and then the crawled URL of the page code to write to the specified file

$urls = Array (' http://www.bkjia.com/', ' http://www.google.com/', ' http://www.example.com/'); Set the page to crawl url$save_to= '/test.txt '; Write the crawled code to the file $st = fopen ($save _to, "a"), $MH = Curl_multi_init (), foreach ($urls as $i = = $url) {$conn [$i] = Curl_init ($u RL); curl_setopt ($conn [$i], Curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0); curl_setopt ($conn [$i], Curlopt_header, 0); curl_setopt ($conn [$i], curlopt_connecttimeout,60); curl_ Setopt ($conn [$i], Curlopt_file, $st); Writes the crawled code to the file Curl_multi_add_handle ($MH, $conn [$i]);} Initialize do {curl_multi_exec ($MH, $active);} while ($active); Execute foreach ($urls as $i + $url) {Curl_multi_remove_handle ($MH, $conn [$i]), Curl_close ($conn [$i]);}//End cleanup Curl_ Multi_close ($MH); fclose ($st);

(2) The code below is almost the same as above, except that this place is where the obtained code is first put into the variable and then the obtained content is written to the specified file

$urls = Array (' http://www.bkjia.com/', ' http://www.google.com/', ' http://www.example.com/'); $save _to= '/test.txt '; Write the crawled code to the file $st = fopen ($save _to, "a"), $MH = Curl_multi_init (), foreach ($urls as $i = = $url) {$conn [$i] = Curl_init ($u RL); curl_setopt ($conn [$i], Curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0); curl_setopt ($conn [$i], Curlopt_header, 0); curl_setopt ($conn [$i], curlopt_connecttimeout,60); curl_ Setopt ($conn [$i],curlopt_returntransfer,true); Instead of writing to the browser, the crawl substitution code is converted to a string Curl_multi_add_handle ($MH, $conn [$i]);} do {curl_multi_exec ($MH, $active),} while ($active), foreach ($urls as $i = = $url) {$data = Curl_multi_getcontent ($conn [ $i]); Gets the code string for the crawl fwrite ($st, $data); Write string to file}//Get data variable and write to file foreach ($urls as $i = = $url) {curl_multi_remove_handle ($MH, $conn [$i]); Curl_close ($conn [ $i]);} Curl_multi_close ($MH); fclose ($st);

(3) The following code is implemented using the PHP Curl Functions implementation of concurrent multi-threaded download files

$urls =array (' http://www.bkjia.com/5w.zip ', ' http://www.bkjia.com/5w.zip ', ' http://www.bkjia.com/5w.zip '); $save _ To= './home/'; $mh =curl_multi_init (); foreach ($urls as $i = = $url) {$g = $save _to.basename ($url); if (!is_file ($g)) {$   conn[$i]=curl_init ($url);   $fp [$i]=fopen ($g, "w"); curl_setopt ($conn [$i],curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0;   Windows NT 6.0) ");   curl_setopt ($conn [$i],curlopt_file, $fp [$i]);   curl_setopt ($conn [$i],curlopt_header, 0);   curl_setopt ($conn [$i],curlopt_connecttimeout,60); Curl_multi_add_handle ($MH, $conn [$i]); }}do{$n =curl_multi_exec ($MH, $active);} while ($active), foreach ($urls as $i = = $url) {curl_multi_remove_handle ($MH, $conn [$i]), Curl_close ($conn [$i]); Fclose ($fp [$i]);} Curl_multi_close ($MH); $urls =array (' http://www.bkjia.com/5w.zip ', ' http://www.bkjia.com/5w.zip ', ' http:// Www.bkjia.com/5w.zip '); $save _to= './home/'; $mh =curl_multi_init (); foreach ($urls as $i = = $url) {$g = $save _ To.basename ($url);   if (!is_file ($g)) {$conn [$i]=curl_init ($url);$fp [$i]=fopen ($g, "w"); curl_setopt ($conn [$i],curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0;   Windows NT 6.0) ");   curl_setopt ($conn [$i],curlopt_file, $fp [$i]);   curl_setopt ($conn [$i],curlopt_header, 0);   curl_setopt ($conn [$i],curlopt_connecttimeout,60); Curl_multi_add_handle ($MH, $conn [$i]); }}do{$n =curl_multi_exec ($MH, $active);} while ($active), foreach ($urls as $i = = $url) {curl_multi_remove_handle ($MH, $conn [$i]), Curl_close ($conn [$i]); Fclose ($fp [$i]);} Curl_multi_close ($MH);

The above mentioned is the whole content of this article, I hope you can like.

http://www.bkjia.com/PHPjc/1029599.html www.bkjia.com true http://www.bkjia.com/PHPjc/1029599.html techarticle PHP with Curl implementation of multi-threaded crawl, Phpcurl multi-threaded crawl PHP with curl implementation of multi-threaded CRAWL php/*curl * * */** * Curl Multithreading * * @param array $array parallel URLs ...

