Design and implementation of a new generation acquisition platform (web crawler)

The next generation of web crawler should meet the following functional characteristics

1. High Performance

2. distributed deployment with unlimited scale expansion

3. flexible configuration, arbitrary rules can be configured

4. support for individual crawler settings: Execution cycle, execution start time, execution interval, and other parameters

5. Support http , HTTPS and other agreements

6. Support Get Way and Post Way

7. support File Download

8. Support page-flipping acquisition

9. support two times JS Request Acquisition

10. Support HTML labels, JSON and so on any message

11. Support Login Capture

12. support proxy collection, with Agent pool function, automatic detection of agent validity at timed time

13. Original rule configuration system with the ability to handle complex messages

14. with the processing capacity of the acquisition results, such as: Replace, intercept, remove the junk label, clear the space and other operations

15. support multi-database, multi-table data results

16. ability to support database insert, delete, modify, etc.

17. Support save file function

18. has a regular failure alarm function

Crawler Server Management

(1) Add, modify, view, and delete crawler servers

(2) control the start-up and stop of distributed crawler servers.

(3) Configure the crawler execution cycle, execution start time, execution interval and other parameters

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6F/A6/wKiom1Wj6gajbXDbAAIv83S6Mtg824.jpg "title=" A.png " alt= "Wkiom1wj6gajbxdbaaiv83s6mtg824.jpg"/>650) this.width=650; src= http://s3.51cto.com/wyfs02/M00/6F/A4/ Wkiol1wj7brqlalkaaezwnefzfu315.jpg "title=" B.png "alt=" Wkiol1wj7brqlalkaaezwnefzfu315.jpg "/>

URL Management (queue management), configuring URL-related Parameters

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6F/A6/wKiom1Wj6mfzr4imAAJYuQhz2Ws326.jpg "title=" E.png " alt= "Wkiom1wj6mfzr4imaajyuqhz2ws326.jpg"/>

Group Management, maintain the group information of the URL, easy to classify and operateConfigure parsing rules

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6F/A4/wKioL1Wj7GrAB4HFAAIv82CtHD8866.jpg "title=" C.png " alt= "Wkiol1wj7grab4hfaaiv82cthd8866.jpg"/>

Proxy service Settings

Proxy Server ImportFailure Rule Alarm

I have been engaged in network research and development for many years, have the network crawler problem need consultation can contact me Q 130 51777 62

A

Design and implementation of a new generation acquisition platform (web crawler)