Php multi-thread webpage capture code sharing

Php multi-thread webpage capture code sharing

This article introduces how to use php to capture web page code with multiple threads. For more information, see. In php, you can use Curl to complete various file transfer operations, such as simulating a browser to send GET or POST requests. The php language itself does not support multithreading, so the efficiency of developing crawler programs is not high. Therefore, Curl Multi Functions is used to implement concurrent Multi-threaded access to multiple URLs. For the basic content of curl, refer to the following article: php curl application instance analysis php curl usage instance code php curl Learning Summary This section describes how to use Curl Multi Functions to concurrently download objects using multiple threads. Example 1: get the content and write it directly to the file $ Url) {$ conn [$ I] = curl_init ($ url); curl_setopt ($ conn [$ I], CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; windows NT 6.0) "); curl_setopt ($ conn [$ I], CURLOPT_HEADER, 0); curl_setopt ($ conn [$ I], CURLOPT_CONNECTTIMEOUT, 60 ); curl_setopt ($ conn [$ I], CURLOPT_FILE, $ st); // write The crawled code to the file curl_multi_add_handle ($ mh, $ conn [$ I]);} // initialize do {curl_multi_exec ($ mh, $ active);} while ($ active); // execute foreach ($ Urls as $ I =>$ url) {curl_multi_remove_handle ($ mh, $ conn [$ I]); curl_close ($ conn [$ I]);} // stop cleaning curl_multi_close ($ mh); fclose ($ st);?> Example 2: Add the retrieved content to the variable and write it to a file $ Url) {$ conn [$ I] = curl_init ($ url); curl_setopt ($ conn [$ I], CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; windows NT 6.0) "); curl_setopt ($ conn [$ I], CURLOPT_HEADER, 0); curl_setopt ($ conn [$ I], CURLOPT_CONNECTTIMEOUT, 60 ); curl_setopt ($ conn [$ I], CURLOPT_RETURNTRANSFER, true); // Set to convert the crawling code to a string without outputting it to the browser curl_multi_add_handle ($ mh, $ conn [$ I]);} do {curl_multi_exec ($ mh, $ active);} while ($ active); f Oreach ($ urls as $ I =>$ url) {$ data = curl_multi_getcontent ($ conn [$ I]); // Obtain The crawled code string fwrite ($ st, $ data); // write a string to a file. It is also possible to store data in the database. } // Get the data variable and write the file foreach ($ urls as $ I =>$ url) {curl_multi_remove_handle ($ mh, $ conn [$ I]); curl_close ($ conn [$ I]);} curl_multi_close ($ mh); fclose ($ st);?>