PHP Crawler most complete summary 2-phpquery,phpcrawer,snoopy frame Chinese Introduction

1. Comparison of several common PHP crawler frameworks

Original link: 52804440

1.1 Phpquery

Advantage: The ability to search the DOM like jquery's powerful.
PQ () is a powerful way to search the DOM, just like jquery's $ (), and the jquery selector is basically available on phpquery, as long as the "." Become "--", the demo is as follows (corresponding to my github Demo5)

<?PHPrequire(' phpquery/phpquery.php '); Phpquery:: Newdocumentfile (' http://www.baidu.com/'); $menu _a= PQ ("a"); foreach($menu _a  as $a){    EchoPq$a)->html (). " <br> "; } foreach($menu _a  as $a){    EchoPq$a)->attr ("href"). " <br> "; } ?>

1.2 Phpcrawer

Advantage: filtration ability is stronger.
The official demo is as follows (my github corresponds to Demo4):

<?PHPinclude("phpcrawl/libs/phpcrawler.class.php"); classMycrawlerextendsPhpcrawler {functionHandledocumentinfo (Phpcrawlerdocumentinfo$PageInfo)       { //As example we just print out the URL of the document        Echo $PageInfo->url. " <br> "; }     }    $crawler=NewMycrawler (); $crawler->seturl ("www.baidu.com"); $crawler->addurlfilterrule ("#\." ( jpg|gif) $# i "); //filter to URLs that contain these image formats    $crawler-go ();?>

1.3 snoopy

Advantages: Submit a form, set up an agent, etc.
Snoopy is a PHP class that simulates the functionality of a browser, can get Web content, send a form,
The demo is as follows (corresponds to DEMO3 in GitHub):

include ' snoopy/snoopy.class.php '; $snoopy New Snoopy (); $url = "http://www.baidu.com"; // $snoopy->fetch ($url);//$snoopy->fetchtext ($url);//Remove HTML tags and other unrelated data $snoopy->fetchform ($url); // only get form//return links in Web pages by default, relative links are automatically completed and converted to full URLs. $snoopy->fetchlinks ($url); Var_dump ($snoopy->results);

1.4 Phpspider

Advantage: Install configuration to Database
Provides the installation configuration, can directly connect the MySQL database, the use is also relatively extensive, here we do not introduce separately.

2. Simulating user Behavior 2.1 file_get_contents

<?PHP$opts=Array(  ' http ' =Array(    ' Method ' = ' GET ', ' header ' = ' accept-language:en\r\n '. "Cookie:foo=bar\r\n"  ));$context=stream_context_create($opts);/*sends an HTTP request for www.example.com with additional headers shown above*/$fp=fopen(' http://www.example.com ', ' R ',false,$context);Fpassthru($fp);fclose($fp);?>

2.2 Curl

$ch=curl_init ();//initializing a Curl sessioncurl_setopt ($ch, Curlopt_url,$url);//set the URL address to get//set the browser's specific headercurl_setopt ($ch, Curlopt_httpheader,Array(  "Host:www.baidu.com", "connection:keep-alive", "accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/* ; q=0.8 "," upgrade-insecure-requests:1 "," dnt:1 "," accept-language:zh-cn,zh;q=0.8,en-gb;q=0.6,en;q=0.4,en-us;q=0.2 "," COOKIE:_ZA=4540D427-EEE1-435A-A533-66ECD8676D7D; "    ));$result=curl_exec ($ch);//perform a Curl session

2.3 Snoopy

• Form submission

One example of us
Form-demo.html

<! DOCTYPE html>        User name : <input type= "text" name= "UserName" ><br>         password : <input type= "password" name= "password" > <br>        <input type= "Submit" >    </form></body>
form-demo.php
<?PHP$userName=$_post[' UserName ']; $password=$_post[' Password ']; if($userName= = = "Admin" &&$password= = = "Admin"){        Echo"Hello admin"; }Else{        Echo"Login Error"; } ?>" submit Form" PHP<?PHPinclude' Snoopy/snoopy.class.php ';$snoopy=NewSnoopy ();$formvars["userName"] = "admin";//UserName is consistent with server-side/Form Name Property$formvars["password"] = "admin";$action= "http://localhost:8000/spider/demo3/form-demo.php";//form submission Address$snoopy->submit ($action,$formvars);Echo $snoopy-results;? ><divclass= "Se-preview-section-delimiter" ></div>
PHP Crawler most complete summary 2-phpquery,phpcrawer,snoopy frame Chinese Introduction