PHP multi-threaded batch collection download pictures
?
Using Curl Multi-threading, in addition to curl can set the request time, encountered a very slow URL resources, can be decisive to give up, so there is no blocking, in addition to multi-threaded request, efficiency should be relatively high, reference: "Curl learning and application [with multithreading]", we come to test it;
Core code:
?
/** * Curl Multithreading * * @param array $array parallel URL * @param int $timeout Time Out * @return Mix */public func tion Curl_http ($array, $timeout = ' + ') {$res = array (); $MH = Curl_multi_init ();//Create Multiple Curl handles foreach ($array as $k = = $url) {$conn [$k]=curl_init ($url);//Initialize Curl _setopt ($conn [$k], curlopt_timeout, $timeout);//Set time-out curl_setopt ($conn [$k], Curlopt_useragent, ' mozilla/5.0 (comp atible; MSIE 5.01; Windows NT 5.0) '); curl_setopt ($conn [$k], Curlopt_maxredirs, 7),//http directional level, 7 maximum curl_setopt ($conn [$k], Curlopt_header, false);//Don't do it here. Ader, add block efficiency curl_setopt ($conn [$k], curlopt_followlocation, 1); 302 redirect Curl_setopt ($conn [$k], curlopt_returntransfer,1);//requires the result to be a string and output to the on-screen curl_setopt ($conn [$k], CURLOPT_HT Tpget, True); Curl_multi_add_handle ($MH, $conn [$k]); }//Prevent death cycle consumes the CPU this paragraph is based on the online notation do {$MRC = Curl_multi_exec ($MH, $active);//When no data, active=true} while ($MRC = = Curlm_call_multi_perfoRM) and//when data is being accepted while ($active and $MRC = = CURLM_OK) {//when there is no data or when a request is paused, active=true if (Curl_multi_select ($MH )! =-1) {do {$MRC = Curl_multi_exec ($MH, $active); } while ($MRC = = Curlm_call_multi_perform); }} foreach ($array as $k = + $url) {if (!curl_errno ($conn [$k])) {$data [$k]=curl_multi_getco Ntent ($conn [$k]);//data is converted to array $header [$k]=curl_getinfo ($conn [$k]);//return HTTP header information Curl_close ($conn [$k]);//Close Handle Curl_multi_remove_handle ($MH, $conn [$k]); Release Resources}else{unset ($k, $url); }} curl_multi_close ($MH); return $data; }//parameter Receive $callback = $_get[' callback ']; $hrefs = $_get[' HREFs ']; $urlarray = explode (', ', trim ($hrefs, ', ')); $date = Date (' Ymd ', Time ()),//Instantiate $img = new Httpimg (), $stime = $img->getmicrotime ();//Start $data = $img->curl_http ($urlarray, ' 20 ');//List Data mkdir ('./img/'. $date, 0777); foreach ((array) $data as $k => $v) {preg_match_all ("/(HREF|SRC) = ([\" | ']?) ([^ \ "' >]+\. (jpg|png| Png| Jpg|gif) \\2/i ", $v, $matches [$k]), if (count ($matches [$k][3]) >0) {$dataimg = $img->curl_http ($matches [$k][3], ' 20 ');//All picture data binary $j = 0;foreach ((array) $dataimg as $kk = + $vv) {if ($VV! = ") {$rand = rand (1000,9999); $basename = time ()." _ ". $rand.". jpg;//files saved in jpg format $fname = './img/'. $date. " /"." $basename "; file_put_contents ($fname, $VV); $j ++;echo" Create section ". $j." Photo "." $fname "."
";} Else{unset ($kk, $VV);}}} Else{unset ($matches);}} $etime = $img->getmicrotime ();//End time echo "Spents". ($etime-$stime). " Seconds "; exit;
?
?
Test the effect.
337 Pictures spents 260 seconds or so, basically can be done within a second can be collected a picture of the effect, and found that the more the image of the advantage of the acquisition speed more obvious.
We can take a look at the file name: You can create 10 images at the same time,
Because of the 20-second request time limit, some pictures generated after the apparent incomplete, that is, the picture resources in 20 seconds failed to fully capture, this time you can set their own.
?