Java crawler Webcollector
Crawler Introduction:
Webcollector is a Java crawler framework (kernel)that requires no configuration and is easy to develop two times,providing a streamlined API that enables a powerful crawler with a small amount of code.
Crawler cores:
Webcollector is committed to maintaining a stable, extensible crawler core, enabling developers to develop flexibly and two times. The kernel is highly extensible, and users can develop their own crawler on the basis of the kernel. The Jsoup is integrated in the source code for accurate page parsing. The selenium is integrated in the 2.x release to handle JavaScript-generated data.
Page Body Extract:
page Body Extract item Contentextractor has been incorporated into Webcollector maintenance.
Webcollector's body Extraction API is encapsulated as a static method of the Contentextractor class. You can extract structured news, or you can just extract the body of the page (or the element in which the body is located).
Body Extract Effect Indicator:
Header extraction and date extraction use a simple heuristic algorithm that is not tested on a standard dataset like the body extraction algorithm, and the algorithm is still being updated.
The Webcollector 1.x version is now transferred to http://git.oschina.net/webcollector/WebCollector-1.x Maintenance and is recommended for use with the 2.x version.
2.x:Webcollector 2.x version features:
2) built-in Berkeley DB Management URL, You can handle a larger number of pages
4) Direct support for multi-agent random switchover
5) Integrated Spring JDBC and MySQL connector for ease of data persistence
6) integrated JSON parser
7) use slf4j as the log façade
8) Modify the HTTP request interface to make it easier for users to customize HTTP requests
Webcollector 2.x website and image:
Official website:Https://github.com/CrawlScript/WebCollector
Image:http://git.oschina.net/webcollector/WebCollector
Webcollector 2.x Tutorial:
Webcollector 2.x tutorial (mirror):
version update information:
webcollector 2.10 released, update details as follows:
1) added news page body automatic extraction algorithm, can automatically extract news page body, title and date, can refer to Example: Webcollector text extract demo.
Webcollector 2.09 released, update details as follows:
1) New Extractor Extractor and crawler multiextractorcrawler for loading the extractor , provide a kind of crawler development program which is convenient for division of labor and maintenance. Refer to Example: Webcollector extractor demo.
2) New filesystemoutput tostore Web pages on site in accordance with the topology structure.
3) Fix HTTPS request certificate issue.
Webcollector is a open source web crawler framework based on Java.it provides some simple interfaces for crawling the web , you can setup a multi-threaded web crawler in less than 5 minutes. -more ...
http://crawlscript.github.io/WebCollector/
| Issues | ||
| #10 | How do I not let the log output to the console | by t0ky04cti0n 2015-08-03 |
| #9 | Can you tell me if the source code of Weibohelper is open? | by Dongshuaike 2015-08-01 |
| #8 | Excuse me, where is the 2.x maven warehouse? | by Fishtrees 2015-06-29 |
| #7 | Do you support proxy? In addition, if supported, whether to support dynamic rotation? | by Rangerwolf 2015-05-05 |
| #5 | Crawl Ajax or JS jump page How to set the policy? | by joe1100 2015-06-06 |
Master Branch code last update: 2015-08-05
Download zipJava crawler Webcollector
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
1 on 1 presale consultation
24/7 Technical Support 6 Free Tickets per Quarter Faster Response
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.
A comprehensive suite of global cloud computing services to power your business
Payment Methods We Support
