Simple crawling based on PHPQuery (2) -- crawling content pages

3. next, the second step is to redirect the url in the list page to the corresponding content page to capture the content to generate a document page in the true sense, because there are many pages generated here, it is slow to capture the page content and generate a local page, so the thought of executing and outputting at the same time is: & lt ;? Phpset_time_limit (0); date_default_timezone_s

3. next, the second step is to redirect the url in the list page to the corresponding content page to capture the content to generate a document page in the true sense, because there are many pages generated here, it is slow to capture the page content and generate a local page. Therefore, the thinking of side execution and output is re-integrated:

 Html (); // The Returned string type variable $ arr = preg_split ("/\ n/", $ ); // use a linefeed to split the string into an array unset ($ arr [0]); // print_r ($ arr); ob_end_clean (); // disable the cache area echo date ('Y-m-d H: I: S', time () before loop output ()). "==> start creating PHPQuery user guide... "; flush (); // push directly to browser foreach ($ arr as $ link) {preg_match ("/(. *) <\/a>/", $ link, $ array); // use the regular expression to match the link and text. // print_r ($ array ); $ https = str_replace ("http", "https", $ array [1]); // It is strange that https processed by phpquery is changed to http. change it back here, otherwise, the object cannot be directed to the document. Content page // echo $ https. ""; $ ch = curl_init (); curl_setopt ($ ch, CURLOPT_URL, $ https); curl_setopt ($ ch, CURLOPT_HEADER, false); curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, true); curl_setopt ($ ch, CURLOPT_SSL_VERIFYPEER, false); curl_setopt ($ ch, CURLOPT_USERAGENT, 'mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) chrome/23.0.20.1.1 Safari/537.11 '); $ res = curl_exec ($ ch); $ rescode = curl_g Etinfo ($ ch, CURLINFO_HTTP_CODE); curl_close ($ ch); phpQuery: new1_enthtml ($ res); $ manual = pq ("# wikicontent")-> html (); // obtain the document content. // echo $ manual. ""; $ title = strtolower ($ array [2]); file_put_contents (". /manual /". $ title. ". html ", $ manual); echo $ title. "The page is created successfully! "; Flush (); sleep (1);} echo date ('Y-m-d H: I: S', time ()). "<== the PHPQuery user guide has been created! ";

. In this way, the corresponding document content page is generated under the Directory/manual.