Open source web crawler and some introduction and comparison

To the current network of open-source web crawler and some introduction and comparison

At present, there are many open-source web crawler on the network for us to use, the best crawler do is certainly Google, but Google released the Spider is a very early version, the following is a few open-source web crawler simple comparison table:

Here we will compare the Nutch, Larbin, Heritrix three reptiles more detailed comparison:

Nutch

Development language: Java

http://lucene.apache.org/nutch/

Brief introduction:

One of the sub-projects of Apache that belongs to the sub-project under the Lucene project.

Nutch is a full web search engine solution based on Lucene, similar to Google, the distributed processing model based on Hadoop guarantees the performance of the system, and the plug-in mechanism like eclipse ensures the system can be customized and easily integrated into its own applications.

Larbin

Development language: C + +

Http://larbin.sourceforge.net/index-eng.html

Brief introduction

Larbin is an open-source web crawler developed by French young Sébastien Ailleret. The purpose of Larbin is to be able to track the URL of the page to expand the crawl and finally provide a wide range of data sources for search engines.

Larbin is just a reptile, that is to say Larbin crawl only Web pages, as to how the parse thing is done by the user himself. In addition, how to store the database and index things larbin is not provided.

Latbin's initial design was also based on a simple but highly configurable principle, so we can see that a simple larbin crawler can get 5 million of pages per day, very efficiently.

Heritrix

Development language: Java

http://crawler.archive.org/

Brief introduction

Comparison with Nutch

and Nutch. Both are Java open source frameworks,Heritrix is an open source product on SourceForge, Nutch is a sub-project of Apache, they are called web crawlers they implement the principle of basic consistency: deep traversal of the resources of the site, to fetch these resources locally, The method used is to analyze every valid URI of the website and submit the HTTP request to obtain the corresponding result, generate the local file and the corresponding log information and so on.

Heritrix is a "archival crawler"--for full, accurate, deep replication of site content. This includes capturing images and other non-textual content. Crawl and store related content. Do not change the content of the page. Re-crawling the same URL is not replaced for the previous. The crawler launches, monitors, and adjusts through the Web user interface, allowing the elastic definition of the URL to get.

The difference between the two:

Nutch only gets and saves the content that can be indexed. Heritrix is the full receipt of the order. Strive to preserve the original page

Nutch can trim the content or convert the content format.

Nutch save content for database optimization format for later indexing; refresh replaces old content. and Heritrix is adding (appending) new content.

Nutch runs and controls from the command line. Heritrix has a Web control management interface.

Nutch's ability to customize is not strong enough, but now there is a certain improvement. The Heritrix can control more parameters.

Heritrix offers features without nutch, a bit of the whole station download flavor. There is no index and no parsing, and even the repeated crawl URLs are not handled very well.

Heritrix is powerful but it's a bit cumbersome to configure.

Open source web crawler and some introduction and comparison