PHP curl_multi_exec () crawl Web content concurrently

PHP curl_multi_exec () crawl Web content concurrently

PHP is a single-threaded language, so in some way the rate is not as Java this multithreaded language, after all, the main aspect is not here. But PHP also has its own multithreading (in fact, concurrency) method--curl_multi_exec ().

We can use Curll to get the content of the page (do not understand curl can find a simple example to see), but if you get multiple pages at the same time, the speed is not ideal, this time curl_multi_exec () can play a role.

Here are some examples of what I'm grabbing from Youku:

function Async_get_url ($url _array, $wait _usec = 0) {if (!is_array ($url _array)) return false;                                                                                              $wait _usec = intval ($wait _usec);    $data = Array ();    $handle = Array ();                                                                                              $running = 0; $MH = Curl_multi_init ();     Multi Curl Handler $i = 0;                                                                                                  foreach ($url _array as $url) {$ch = Curl_init ();        curl_setopt ($ch, Curlopt_url, $url); curl_setopt ($ch, Curlopt_returntransfer, 1);        Return don ' t print curl_setopt ($ch, Curlopt_timeout, 30); curl_setopt ($ch, Curlopt_useragent, ' mozilla/4.0 (Compatible MSIE 5.01;        Windows NT 5.0) '); curl_setopt ($ch, curlopt_followlocation, 1);                                                                                                  302 redirect Curl_setopt ($ch, Curlopt_maxredirs, 7); Curl_multi_add_handle ($MH, $ch);                                                                                                  Put curl resource into multi curl handler    $handle [$i + +] = $ch; }/* This will avoid CPU loading 10                                                                                              0% questions *//From: http://www.hengss.com/xueyuan/sort0362/php/info-36963.html    do {$MRC = Curl_multi_exec ($MH, $active);                                                                                              } while ($MRC = = Curlm_call_multi_perform); while ($active and $MRC = = CURLM_OK) {if (Curl_multi_select ($MH)! =-1) {do {$MRC = Curl_multi_exec ($MH, $active);        } while ($MRC = = Curlm_call_multi_perform); }}/*//Thanks to the practice of Ren point.    (need to test)//The return value of the curl_multi_exec is used to return the fault of the multi-thread, normally, the return value is 0, which means that only $MRC is used to capture the return value when the recursive loop will only run once, and when it really happens, there is a $mrc sentence that turns the loop.    The function of Curl_multi_select is that Curl sends the request and waits for the status before the response, so it doesn't need to be directed into the empty loop, it's like it's going to judge itself & decide to wait for the time of Sleep ().        /* Read information */foreach ($handle as $i + = $ch) {$content = Curl_multi_getcontent ($ch); $data [$i] = (Curl_errno ($ch) = = 0)?    $content: false; }/* Remove handle*/foreach ($    Handle as $ch) {Curl_multi_remove_handle ($MH, $ch);                                                                                              } curl_multi_close ($MH);                                   return $data;}                                                       $url = "http://m.youku.com/wap/"; $reg 1 = "/(. *?) <\/a>/i ";//Get video link $reg2="/]*) \s*class=\ "imgdetail\" \s*src= (' |\ ") ([^ ' \"]+) (' |\ ')/i "; $reg 3=" "; $reg 4 ="/
 
  
. *?<\/p>/i ";//Get video title (optional)/ /Create two curl Resources $ch1 = Curl_init (); $resultArray =array ();//load All data array $ch=array (),//$ch 2 = Curl_init ();//Specify URL and appropriate parameters Curl_ Setopt ($ch 1, Curlopt_url, $url); curl_setopt ($ch 1, Curlopt_returntransfer, 1); curl_setopt ($ch 1, curlopt_header, 0); $ Content=curl_exec ($ch 1); Curl_close ($ch 1);//$content =file_get_contents ($url);p Reg_match_all ($reg 1, $content, $ matches); $video = $matches [0];//home video link//print_r ($video); foreach ($video as $a + = $key) {$position =strpos ($key, "    href ");    $substring =substr ($key, $position +11);    $pos =strpos ($substring, ">");    $link =substr ($substring, 0, $pos-1); $NEXTURL [$a]= $url. $link;} $url _array = Array (//' http://www.google.com ',//' http://www.baidu.com ',//);//print_r ($NEXTURL);//print_r (Asyn C_get_url ($NEXTURL));//Get the contents of all pages $alldata=async_get_url ($NEXTURL); foreach ($allData as $page) {//Get a video picture Preg_match    _all ($reg 2, $page, $img); $iMg_arr= $img [0];        foreach ($img _arr as $arr) {$position =strpos ($arr, "src");        $sub =substr ($arr, $position +5);        $pos =strpos ($sub, "\" ");    $last =substr ($sub, 0, $pos);    }//Get video HD On Demand address Preg_match_all ($reg 3, $page, $vids);    $video _arr= $vids [0];    $vid = $video _arr[0];    $position =strpos ($vid, "href");    $v _string=substr ($vid, $position +11);    $pos =strpos ($v _string, "\" ");    $add =substr ($v _string, 0, $pos);    $video _url= $url. $add;    Get the title of the video Preg_match_all ($reg 4, $page, $match);    $title = $match [0];    Print_r ($er);    $r =serialize ($title); $position =mb_strpos ($r, "
  
");    $sub =substr ($r, 0, $position);    $pos =mb_strrpos ($sub, ">");                                                                                                  $til =substr ($sub, $pos + 1);    Integrated into an array of $subArray =array (' image ' = $last, ' video ' + $video _url, ' title ' and ' $til '); Array_push ($resultArray, $subArray);} echo Json_encode ($resultArray);
 

Focus on the function with Async_get_url

do {        $MRC = curl_multi_exec ($MH, $active);    } while ($MRC = = curlm_call_multi_perform);          while ($active and $MRC = = CURLM_OK) {        if (Curl_multi_select ($MH)! =-1) {do            {                $MRC = curl_multi_exec ($MH, $a ctive);            } while ($MRC = = Curlm_call_multi_perform);}    }

The above paragraph is also a difficult point.

The first loop, $MRC = = Curlm_call_multi_perform (-1) indicates that the handle resource has not been processed and continues $MRC = Curl_multi_exec ($MH, $active)

In particular, $MRC and $active are all integer types;

When $mrc== curlm_ok (0), it shows that there are still resources, but not yet arrived.

This is on the second loop:

(while) if resources have not arrived

If there is an active connection in the curl batch connection-that is, the handle is dry (refer to the PHP manual for details)

(do-while) Handling handle Resources

Curl concurrency processing Because the official documents are relatively concise, I have also checked a lot of English documents Cai Lue understand.

I hope we can make progress together!

Reference Documentation:

Http://technosophos.com/content/php-and-curlmultiexec

http://blog.longwin.com.tw/2009/10/php-multi-thread-curl-2009/

