Web page crawling: Summary of Web Page crawling in PHP, crawling Crawlers

Source: Internet
Author: User

Web page crawling: Summary of Web Page crawling in PHP, crawling Crawlers

Source: http://www.ido321.com/1158.html


To capture the content of a webpage, You need to parse the DOM tree, find the specified node, and then capture the content we need. This process is a bit cumbersome. LZ summarizes several common and easy-to-implement web page capturing methods. If you are familiar with JQuery selector, these frameworks will be quite simple.

1. Ganon

Project address: http://code.google.com/p/ganon/

Document: http://code.google.com/p/ganon/w/list

Test: capture all the class attribute values on the homepage of my website as the div element of focus and output the class value.

<?php include 'ganon.php'; $html = file_get_dom('http://www.ido321.com/'); foreach($html('div[class="focus"]') as $element) {   echo $element->class, "<br>\n";  }?>

 

Result:

Ii. phpQuery

Project address: http://code.google.com/p/phpquery/

Document: https://code.google.com/p/phpquery/wiki/Manual

Test: capture the article Tag Element on the homepage of my website and publish the html value of the h2 tag

<?phpinclude 'phpQuery/phpQuery.php'; phpQuery::newDocumentFile('http://www.ido321.com/'); $artlist = pq("article"); foreach($artlist as $title){    echo pq($title)->find('h2')->html()."<br/>"; } ?>

 

Result:

3. Simple-Html-Dom

Address: http://simplehtmldom.sourceforge.net/
Document: http://simplehtmldom.sourceforge.net/manual.htm

Test: capture all links on the home page of my website

<? Phpinclude 'simple _ html_dom.php '; // you can use url and file to create DOM $ html = file_get_html ('HTTP: // www.ido321.com /'); // find all images // foreach ($ html-> find ('img ') as $ element) // echo $ element-> src. '<br>'; // find all links, foreach ($ html-> find ('A') as $ element) echo $ element-> href. '<br>';?>

 

Result: (Part 1)

Iv. Snoopy

Project address: http://code.google.com/p/phpquery/

Document: http://code.google.com/p/phpquery/wiki/Manual

Test: capture the homepage of my website

<? Phpinclude ("Snoopy. class. php "); $ url =" http://www.ido321.com "; $ snoopy = new Snoopy; $ snoopy-> fetch ($ url); // get all content echo $ snoopy-> results; // display the result // echo $ snoopy-> fetchtext; // obtain the text content (remove the html code) // echo $ snoopy-> fetchlinks ($ url ); // obtain the link // $ snoopy-> fetchform; // obtain the form?>

 

Result:

5. Manually write Crawlers

If the writing capability is OK, you can write a Web Crawler to capture webpages. LZ will not repeat this article on the Internet. If you are interested, you can crawl it on the Baidu php webpage.

Ps: Resource Sharing

Common open-source crawler projects please stamp: http://blog.chinaunix.net/uid-22414998-id-3774291.html


Next article: the "fart theory" of the National Father-in-law"




Php Web Crawlers collect part of a website

You can use the simpl_html_dom class to collect data. If you have jquery, I believe you will understand it. Good luck.
 
Crawlers crawl web keywords and summaries for search

Strip_tags ($ string)

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.