PHP multi-thread web page capture implementation code _ PHP Tutorial-php Tutorial

PHP multi-threaded web page capture implementation code. Restricted by the fact that the php language does not support multithreading, the efficiency of developing crawler programs is not high. at this time, you often need to use CurlMultiFunctions to implement concurrent and multi-threaded access. due to the fact that the php language itself does not support multithreading, therefore, the efficiency of developing crawler programs is not high. at this time, you often need to use Curl Multi Functions to implement concurrent and Multi-threaded access to multiple url addresses. Since Curl Multi Function is so powerful, can I use Curl Multi Functions to write concurrent Multi-threaded download files? of course, the following code is provided:

Code 1: Write the obtained code directly to a file

The code is as follows:

$ Urls = array (
'Http: // www.sina.com.cn /',
'Http: // www.sohu.com /',
'Http: // www.163.com /'
); // Set the URL of the page to be crawled

$ Save_to = '/test.txt'; // write the captured code to this file

$ St = fopen ($ save_to, "");
$ Mh = curl_multi_init ();

Foreach ($ urls as $ I =>$ url ){
$ Conn [$ I] = curl_init ($ url );
Curl_setopt ($ conn [$ I], CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0 )");
Curl_setopt ($ conn [$ I], CURLOPT_HEADER, 0 );
Curl_setopt ($ conn [$ I], CURLOPT_CONNECTTIMEOUT, 60 );
Curl_setopt ($ conn [$ I], CURLOPT_FILE, $ st); // you can write the crawled code to a file.
Curl_multi_add_handle ($ mh, $ conn [$ I]);
} // Initialization

Do {
Curl_multi_exec ($ mh, $ active );
} While ($ active); // execute

Foreach ($ urls as $ I =>$ url ){
Curl_multi_remove_handle ($ mh, $ conn [$ I]);
Curl_close ($ conn [$ I]);
} // End cleaning

Curl_multi_close ($ mh );
Fclose ($ st );
?>

Code 2: Put the obtained code into a variable before writing it to a file.

The code is as follows:

$ Urls = array (
'Http: // www.sina.com.cn /',
'Http: // www.sohu.com /',
'Http: // www.163.com /'
);

$ Save_to = '/test.txt'; // write the captured code to this file
$ St = fopen ($ save_to, "");

$ Mh = curl_multi_init ();
Foreach ($ urls as $ I =>$ url ){
$ Conn [$ I] = curl_init ($ url );
Curl_setopt ($ conn [$ I], CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0 )");
Curl_setopt ($ conn [$ I], CURLOPT_HEADER, 0 );
Curl_setopt ($ conn [$ I], CURLOPT_CONNECTTIMEOUT, 60 );
Curl_setopt ($ conn [$ I], CURLOPT_RETURNTRANSFER, true); // you can convert the crawler code into a string instead of writing it to the browser.
Curl_multi_add_handle ($ mh, $ conn [$ I]);
}

Do {
Curl_multi_exec ($ mh, $ active );
} While ($ active );

Foreach ($ urls as $ I =>$ url ){
$ Data = curl_multi_getcontent ($ conn [$ I]); // Obtain The crawled code string
Fwrite ($ st, $ data); // write a string to a file. Of course, you can also not write files, such as saving them to the database.
} // Get the data variable and write it to the file

Foreach ($ urls as $ I =>$ url ){
Curl_multi_remove_handle ($ mh, $ conn [$ I]);
Curl_close ($ conn [$ I]);
}

Curl_multi_close ($ mh );
Fclose ($ st );
?>

Pipeline needs to use Curl Multi Functions to implement concurrent Multi-thread access to multiple...