PHP using curl to implement multithreading crawl Web _php skills

PHP using Curl functions can complete a variety of transfer file operations, such as Analog browser to send Get,post request, etc., limited by the PHP language itself does not support multithreading, so the development of the crawler program is not efficient, this time often need to use the Curl Multi Functions it can realize concurrent multithreading access to multiple URL addresses. Since the Curl Multi function is so powerful, can you use Curl Multi functions to write concurrent multi-threaded download files, of course, here are my code:

Code 1: Write the obtained code directly to a file

<?php 
$urls = Array (  
 ' http://www.sina.com.cn/', '  
 http://www.sohu.com/', '  
 http://www.163.com/') 
); Set the page URL to crawl  
   
$save _to= '/test.txt ';  The crawled code is written to the file   
  
$st = fopen ($save _to, "a");  
$MH = Curl_multi_init ();   
  
foreach ($urls as $i => $url) {  
 $conn [$i] = Curl_init ($url);  
 curl_setopt ($conn [$i], Curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) ");  
 curl_setopt ($conn [$i], Curlopt_header, 0);  
 curl_setopt ($conn [$i], curlopt_connecttimeout,60);  
 curl_setopt ($conn [$i], Curlopt_file, $st); Set to write crawled code to file  
 curl_multi_add_handle ($MH, $conn [$i]);  
//Initialize do  
   
{  
 curl_multi_exec ($MH, $ Active);  
} while ($active); Execute  
   
foreach ($urls as $i => $url) {  
 curl_multi_remove_handle ($mh, $conn [$i]);  
 Curl_close ($conn [$i]);  
} End cleaning up  
   
curl_multi_close ($MH);  
Fclose ($st); 
? > 

Code 2: Put the obtained code first into a variable and then write to a file

<?php $urls = Array (' http://www.sina.com.cn/', ' http://www.sohu.com/', ' http://www.163.com/');  $save _to= '/test.txt ';  
  
The crawled code is written to the file $st = fopen ($save _to, "a");  
$MH = Curl_multi_init ();  
 foreach ($urls as $i => $url) {$conn [$i] = Curl_init ($url); curl_setopt ($conn [$i], Curlopt_useragent, "mozilla/4.0 (compatible; MSIE 7.0;  
 Windows NT 6.0) ");  
 curl_setopt ($conn [$i], Curlopt_header, 0);  
 curl_setopt ($conn [$i], curlopt_connecttimeout,60); curl_setopt ($conn [$i],curlopt_returntransfer,true);  
Setting does not write the crawl substitution code to the browser, but converts it to a string Curl_multi_add_handle ($MH, $conn [$i]);  
do {curl_multi_exec ($MH, $active);  
   
while ($active); foreach ($urls as $i => $url) {$data = Curl_multi_getcontent ($conn [$i]);//Get Crawled Code string fwrite ($st, $data);//character String to write to the file. Of course, you can also not write files, such as in the database, and/or get data variables, and write to the file foreach ($urls as $i => $url) {curl_multi_remove_handle ($MH, $conn [$i])  
 ;  
Curl_close ($conn [$i]);  
} curl_multi_close ($MH); Fclose ($ST);  ?>

The above is the entire contents of this article, I hope you can enjoy.