php多線程抓取資訊測試例子

PHP 5.3 以上版本，使用pthreads PHP擴充，可以使PHP真正地支援多線程。多線程在處理重複性的迴圈任務，能夠大大縮短程式執行時間。

PHP擴充下載：https://github.com/krakjoe/pthreads
PHP手冊文檔：http://php.net/manual/zh/book.pthreads.php

1、擴充的編譯安裝(Linux），編輯參數 --enable-maintainer-zts 是必選項：

cd /Data/tgz/php-5.3.8
./configure --prefix=/Data/apps/php --with-config-file-path=/Data/apps/php/etc --with-mysql=/Data/apps/mysql --with-mysqli=/Data/apps/mysql/bin/mysql_config --with-iconv-dir --with-freetype-dir=/Data/apps/libs --with-jpeg-dir=/Data/apps/libs --with-png-dir=/Data/apps/libs --with-zlib --with-libxml-dir=/usr --enable-xml --disable-rpath --enable-bcmath --enable-shmop --enable-sysvsem --enable-inline-optimization --with-curl --enable-mbregex --enable-fpm --enable-mbstring --with-mcrypt=/Data/apps/libs --with-gd --enable-gd-native-ttf --with-openssl --with-mhash --enable-pcntl --enable-sockets --with-xmlrpc --enable-zip --enable-soap --enable-opcache --with-pdo-mysql --enable-maintainer-zts
make clean
make
make install       
 
unzip pthreads-master.zip
cd pthreads-master
/Data/apps/php/bin/phpize
./configure --with-php-config=/Data/apps/php/bin/php-config
make
make install

添加擴充：

 vi /Data/apps/php/etc/php.ini

 extension = "pthreads.so"

一段PHP多線程、與For迴圈，抓取百度搜尋網頁面的PHP程式碼範例：

代碼如下

複製代碼

<?php    class test_thread_run extends Thread     {        public $url;        public $data;          public function __construct($url)        {            $this->url = $url;        }          public function run()        {            if(($url = $this->url))            {                $this->data = model_http_curl_get($url);            }        }    }      function model_thread_result_get($urls_array)     {        foreach ($urls_array as $key => $value)         {            $thread_array[$key] = new test_thread_run($value["url"]);            $thread_array[$key]->start();        }          foreach ($thread_array as $thread_array_key => $thread_array_value)         {            while($thread_array[$thread_array_key]->isRunning())            {                usleep(10);            }            if($thread_array[$thread_array_key]->join())            {                $variable_data[$thread_array_key] = $thread_array[$thread_array_key]->data;            }        }        return $variable_data;    }      function model_http_curl_get($url,$userAgent="")     {        $userAgent = $userAgent ? $userAgent : 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)';         $curl = curl_init();        curl_setopt($curl, CURLOPT_URL, $url);        curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);        curl_setopt($curl, CURLOPT_TIMEOUT, 5);        curl_setopt($curl, CURLOPT_USERAGENT, $userAgent);        $result = curl_exec($curl);        curl_close($curl);        return $result;    }      for ($i=0; $i < 100; $i++)     {         $urls_array[] = array("name" => "baidu", "url" => "http://www.baidu.com/s?wd=".mt_rand(10000,20000));    }      $t = microtime(true);    $result = model_thread_result_get($urls_array);    $e = microtime(true);    echo "多線程：".($e-$t)."n";      $t = microtime(true);    foreach ($urls_array as $key => $value)     {        $result_new[$key] = model_http_curl_get($value["url"]);    }    $e = microtime(true);    echo "For迴圈：".($e-$t)."n";  ?>

例子，採集資料

 

代碼如下

複製代碼

<?php $urls = array( 'http://www.111cn.net/', 'http://www.sohu.com/', 'http://www.163.com/' ); $save_to='/test.txt'; // 把抓取的代碼寫入該檔案 $st = fopen($save_to,"a"); $mh = curl_multi_init(); foreach ($urls as $i => $url) { $conn[$i] = curl_init($url); curl_setopt($conn[$i], CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"); curl_setopt($conn[$i], CURLOPT_HEADER ,0); curl_setopt($conn[$i], CURLOPT_CONNECTTIMEOUT,60); curl_setopt($conn[$i],CURLOPT_RETURNTRANSFER,true); // 設定不將爬取代碼寫到瀏覽器，而是轉化為字串 curl_multi_add_handle ($mh,$conn[$i]); } do { curl_multi_exec($mh,$active); } while ($active); foreach ($urls as $i => $url) { $data = curl_multi_getcontent($conn[$i]); // 獲得爬取的代碼字串 fwrite($st,$data); // 將字串寫入檔案。當然，也可以不寫入檔案，比如存入資料庫 } // 獲得資料變數，並寫入檔案 foreach ($urls as $i => $url) { curl_multi_remove_handle($mh,$conn[$i]); curl_close($conn[$i]); } curl_multi_close($mh); fclose($st); ?>