About PHP web crawler phpspider.

A few days ago, was pulled by the boss told me to crawl the public comment on the data of a store, of course, I was the words of the refusal of righteousness, the reason is I do not ... But my resistance and no egg use, so still obediently to check the information, because I am engaged in PHP work, the first to find is PHP web crawler source, in my unremitting efforts, finally found Phpspider, open phpspider Development document home I was shocked, the title of " I used a reptile to "steal" a day. 1 million users, just to prove that PHP is the best language in the world, as I expected, PHP is the best language in the world. Less nonsense, the following begins to learn to use.

The first thing to look at is a demo, the code is as follows:

$configs=Array(    ' Name ' = ' embarrassing encyclopedia ', ' domains ' =Array(        ' qiushibaike.com ', ' www.qiushibaike.com '    ), ' scan_urls ' =Array(        ' http://www.qiushibaike.com/'    ), ' content_url_regexes ' =Array(        "Http://www.qiushibaike.com/article/\d+"    ), ' list_url_regexes ' =Array(        "Http://www.qiushibaike.com/8hr/page/\d+\?s=\d+"    ), ' fields ' =Array(        Array(            //Extract the contents of a content page' Name ' = ' article_content ', ' selector ' = '//*[@id = ' Single-next-link '] ", ' Required ' =true        ),Array(            //author of the article Extracting content pages' Name ' = ' article_author ', ' selector ' = '//div[contains ' (@class, ' author ')]//h2 ", ' Required ' = >true        ),    ),);$spider=NewPhpspider ($configs);$spider->start ();

Each specific information, you can go to https://doc.phpspider.org/demo-start.html view, where more detailed, here is only the detour I go,

Domains is a domain name that is defined and collected only under that domain name.

Content_url_regexes is a collection of content pages, use Chrome to view the source of the Web page, and then use the selector selector positioning, selector use the XPath format to locate parameters, of course, you can also use CSS to choose.

List_url_regexes the list page, each list page captures multiple content_url_regexes URL loops.
There are some other parameters that are not listed, such as:

' Max_try ' = 5,   array(        ' type ' = ' = ' db ',         array(            ' host '  = ' localhost ',            ' port '  = ' 3306 ',            ' user '  = ' root ',            ' pass '  = ' root ', '            name '  = ' demo ',        ),        ' table    ' = ' 360ky ',

Max_try the number of crawler tasks that work at the same time.

Export acquisition data storage, there are two formats, one is written to the database, one is directly generated. csv format files.
As long as the URL rules are written correctly, they can be run without the package inside the framework. Of course, this framework can only be run under the PHP-CLI command line, so configure the environment variable before use, or the CD to the PHP installation path to run.
Finally, we successfully collected more than 1000 data from a certain point in the mass review.

About PHP web crawler phpspider.