自訂HTTP抓包和過濾

定義一個http抓包類，發送資料到一個自訂的接受指令碼，可以發送成功，並收取資料，但是發送到外網，卻不行，分析過在瀏覽器下發送HTTP請求時的request header 資訊，通過類比請求，但逾時...

//定義一個HTTP抓包類，其實也可以用curl。。。。。hostInfo=parse_url($url);$this->setRequestHeader(array('Host' => $this->hostInfo['host']));$this->setRequestHeader(array('Connection' => 'keep-alive'));}//設定HTTP請求行資訊，例如： GET /resources  HTTP/1.1//但為了避免漏掉url中?開始的查詢資訊，有必要進行判斷public function setRequestLine($method){//如果是POST請求，則自動添加content-type頭資訊if(strtolower($method)=='post'){$this->setRequestHeader(array('Content-type' => 'application/x-www-form-urlencoded'));}if(!empty($this->hostInfo['query'])){$this->requestLine=strtoupper($method)."  ".$this->hostInfo['path']."?".$this->hostInfo['query']."  HTTP/1.1 \r\n";}else{$this->requestLine=strtoupper($method)."  ".$this->hostInfo['path']."  HTTP/1.1 \r\n";}}//設定HTTP要求標頭。//接收參數是數群組類型，通過迭代拼接key:value，並換行public function setRequestHeader($header){foreach($header as $key => $value){$this->requestHeader .=$key.":".$value."\r\n";}}//設定HTTP請求體//接收參數是數群組類型，通過迭代拼接key=value,因為最後一席拼接會有一個多餘的&，所以有必要去掉public function setRequestBody($body){foreach($body as $key => $value){$this->requestBody .=$key.'='.$value.'&';}$offset=strrpos($this->requestBody, '&');$this->requestBody=substr($this->requestBody, 0, $offset);}//組裝  請求行+要求標頭+請求體，並根據請求體的長度，自動填滿要求標頭的content-length欄位public function setRequestEntity(){if(!empty($this->requestBody)){$contentLength=strlen($this->requestBody);$this->setRequestHeader(array('Content-length' => $contentLength));$this->requestEntity=$this->requestLine.$this->requestHeader."\r\n".$this->requestBody;}else{$this->requestEntity=$this->requestLine.$this->requestHeader."\r\n";}}//解析主機名稱的函數，暫時沒有用上.......public function parseHost($url){$pat='#http://([^/]+)#i';if(preg_match($pat, $url, $match)){return $match[1];}else{echo '匹配主機資訊失敗
';}}//建立到主機的串連public function createConnect(){$this->connect=fsockopen($this->hostInfo['host'], 80, $this->errNo, $this->errStr) or die('串連主機失敗'.$this->errStr);}//發送請求public function sendRequest(){$this->setRequestEntity();echo $this->requestEntity;exit();$this->createConnect();$entityLength=strlen($this->requestEntity);if($entityLength != fwrite($this->connect, $this->requestEntity, $entityLength)){die('寫入資料失敗
');}else{$this->receiveResponse();}}//接受請求，並依次拼接響應體public function receiveResponse(){while(!feof($this->connect)){$this->responseEntity .= fread($this->connect, 1024);}}//計算回應標頭與響應體之間的空行的位置public function calculateEmptyLinePos(){$this->emptyLinePos=strpos($this->responseEntity,"\r\n\r\n",0);}//接受響應體的頭部....public function receiveResponseHeader(){$this->calculateEmptyLinePos();$this->responseHeader=substr($this->responseEntity, 0, $this->emptyLinePos);echo $this->responseHeader;}//接收響應體的body部分public function receiveResponseBody(){$this->calculateEmptyLinePos();$this->responseBody=substr($this->responseEntity, $this->emptyLinePos);}//返回請求結果public function getResponse(){return $this->responseEntity;}public function parseResponse(){}public function __destruct(){//fclose($this->connect);} }set_time_limit(60);$http=new Httpwrap("http://www.mmkao.com/Beautyleg/");//設定HTTP請求行$http->setRequestLine("get");//設定HTTP頭$http->setRequestHeader(array("Accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"));$http->setRequestHeader(array("Accept-Language" => "zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3"));$http->setRequestHeader(array("Accept-Encoding" => "gzip, deflate"));$http->setRequestHeader(array("User-Agent" => "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36"));//$http->setRequestHeader(array("Cookie" => "BAIDU_DUP_lcr=http://www.baidu.com/s?wd=beautyleg&rsv_spt=1&issp=1&f=3&rsv_bp=0&rsv_idx=2&ie=utf-8&tn=baiduhome_pg&rsv_enter=1&rsv_sug3=6&rsv_sug4=415&rsv_sug1=3&oq=beauty&rsv_sug2=0&rsp=0&inputT=2363; safedog-flow-item=8471BA510DA33350ED344AC374D3044A; bdshare_firstime=1415165097782; cscpvrich_fidx=6; AJSTAT_ok_pages=2; AJSTAT_ok_times=2; CNZZDATA3811623=cnzz_eid%3D253823549-1415164312-http%253A%252F%252Fwww.baidu.com%252F%26ntime%3D1415169712"));//發送資料$http->sendRequest();//$http->receiveResponseHeader();?>通過這個類給領一個自訂的指令碼，可以發送和接收資料，領一個指令碼如下：但是給這個網站發送請求時，卻逾時：網站是：http://www.mmkao.com/Beautyleg/通過chrome給這個網站首頁發送請求時的header頭資訊：Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8Accept-Encoding:gzip,deflate,sdchAccept-Language:zh,en;q=0.8,zh-TW;q=0.6,zh-CN;q=0.4,ja;q=0.2Cache-Control:max-age=0Connection:keep-aliveCookie:BAIDU_DUP_lcr=http://www.baidu.com/s?wd=beautyleg&rsv_spt=1&issp=1&f=3&rsv_bp=0&rsv_idx=2&ie=utf-8&tn=baiduhome_pg&rsv_enter=1&rsv_sug3=6&rsv_sug4=415&rsv_sug1=3&oq=beauty&rsv_sug2=0&rsp=0&inputT=2363; safedog-flow-item=8471BA510DA33350ED344AC374D3044A; bdshare_firstime=1415165097782; cscpvrich_fidx=7; AJSTAT_ok_pages=3; AJSTAT_ok_times=2; CNZZDATA3811623=cnzz_eid%3D253823549-1415164312-http%253A%252F%252Fwww.baidu.com%252F%26ntime%3D1415169712DNT:1Host:www.mmkao.comUser-Agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36Response Headersview source//通過相同的封裝，並調用Httpwrap發送請求時，卻提示逾時，是在不知道哪裡出問題........針對這個網站寫了一個過濾出圖片連結的類：responseBody=$body;}//匹配圖片src開始的連結地址public function feedImage(){$pat='#imgLink[]=$link;}}else{echo '匹配失敗圖片連結地址失敗'."
";}}//提取head部分public function filterHeader($body){$pat='#[\s\S]+#im';if(preg_match($pat, $body, $match)){$this->header=$match[0];}else{echo '匹配head部分失敗'."
";}}//提取body部分public function filterBody($body){$pat='#[\s\S]+#im';if(preg_match($pat, $body, $match)){$this->body=$match[0];}else{echo '匹配body部分失敗'."
";}}//提取分頁資訊，這個只能針對性的匹配，不能通用public function rollPage($body){$pat='#[\x{4e00}-\x{9fa5}]+\s*\d\s+?/\s+?\d+\s*[\x{4e00}-\x{9fa5}]*#ui';if(preg_match($pat, $body, $match)){$patNum='#/\s*(\d\d*)#';if(preg_match($patNum, $match[0], $num)){$this->pageNum=$num[1];}else{echo '提取分頁具體值失敗'."
";}}else{echo '提取分頁統計失敗'."
";}}?>附註：  這兩個類，，都通過了內網的測試，並成功過濾出圖片的連結，但是給http://www.mmkao.com/Beautyleg/發送請求時，卻提示逾時，，不知道哪裡出了問題。。。。。。

回複討論(解決方案)

在window的命令列下，提交請求，是可以收到資料的......


可以收到資料

$url = 'http://www.mmkao.com/Beautyleg/';print_r(get_headers($url));

Array(    [0] => HTTP/1.1 200 OK    [1] => Connection: close    [2] => Date: Wed, 05 Nov 2014 08:53:09 GMT    [3] => Content-Length: 13889    [4] => Content-Type: text/html    [5] => Content-Location: http://www.mmkao.com/Beautyleg/index.html    [6] => Last-Modified: Wed, 05 Nov 2014 05:39:09 GMT    [7] => Accept-Ranges: bytes    [8] => ETag: "e8939ad2baf8cf1:693"    [9] => Server: IIS    [10] => X-Powered-By: WAF/2.0    [11] => Set-Cookie: safedog-flow-item=8471BA510DA33350ED344AC374D3044A; expires=Sat, 12-Dec-2150 10:26:25 GMT; domain=mmkao.com; path=/)

$url = 'http://www.mmkao.com/Beautyleg/';print_r(get_headers($url));

Array(    [0] => HTTP/1.1 200 OK    [1] => Connection: close    [2] => Date: Wed, 05 Nov 2014 08:53:09 GMT    [3] => Content-Length: 13889    [4] => Content-Type: text/html    [5] => Content-Location: http://www.mmkao.com/Beautyleg/index.html    [6] => Last-Modified: Wed, 05 Nov 2014 05:39:09 GMT    [7] => Accept-Ranges: bytes    [8] => ETag: "e8939ad2baf8cf1:693"    [9] => Server: IIS    [10] => X-Powered-By: WAF/2.0    [11] => Set-Cookie: safedog-flow-item=8471BA510DA33350ED344AC374D3044A; expires=Sat, 12-Dec-2150 10:26:25 GMT; domain=mmkao.com; path=/)

這樣確實可以。蛋疼，我再看看

GET /Beautyleg/ HTTP/1.1
Host:www.mmkao.com
Connection:keep-alive
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language:zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding:gzip, deflate

HTTP/1.1 200 OKDate: Wed, 05 Nov 2014 09:34:02 GMTContent-Length: 13889Content-Type: text/htmlContent-Location: http://www.mmkao.com/Beautyleg/index.htmlLast-Modified: Wed, 05 Nov 2014 05:39:09 GMTAccept-Ranges: bytesETag: "e8939ad2baf8cf1:693"Server: IISX-Powered-By: WAF/2.0Set-Cookie: safedog-flow-item=BFC86EA38C3E0337FB45DCE403130335; expires=Sat, 12-Dec-2150 11:07:18 GMT; domain=mmkao.com; path=/Beautyleg_咪咪圖秀............................                
 

                          共 1035 組，每頁 30 組，當前 1 / 35 頁。首頁 上一頁 下一頁 尾頁。 轉到第                                                                                                                                                                                                                                                                                     咪咪圖秀 www.mmkao.com                 這個是我通過Httpwrap發送的要求標頭的資訊，複製到telnent上也能完整擷取網頁                   $url = 'http://www.mmkao.com/Beautyleg/';print_r(get_headers($url));   Array(    [0] => HTTP/1.1 200 OK    [1] => Connection: close    [2] => Date: Wed, 05 Nov 2014 08:53:09 GMT    [3] => Content-Length: 13889    [4] => Content-Type: text/html    [5] => Content-Location: http://www.mmkao.com/Beautyleg/index.html    [6] => Last-Modified: Wed, 05 Nov 2014 05:39:09 GMT    [7] => Accept-Ranges: bytes    [8] => ETag: "e8939ad2baf8cf1:693"    [9] => Server: IIS    [10] => X-Powered-By: WAF/2.0    [11] => Set-Cookie: safedog-flow-item=8471BA510DA33350ED344AC374D3044A; expires=Sat, 12-Dec-2150 10:26:25 GMT; domain=mmkao.com; path=/)         這個問題遺留了好久，今天花時間解決了，而且可以整站採集,,,,,   http://blog.csdn.net/free_program_1314/article/details/41780835