Day 1: Project Construction, Day 1 Project Construction

Day 1: Project Construction, Day 1 Project Construction

I. Preface

1.0. Since I have not written about the general crawler framework, I searched around and found a lot of information. Here I used the framework model introduced by a great god. I forgot the reference address, so I will not post it here.

2.0 the previously mentioned Verification Code module also stopped, and then we will talk about it together in this classification.

Ii. Text

2.1. Frame Design Diagram

　　

2.2. the figure above shows the relationship between each module. I copied and pasted the specific description.

①, Scheduler:Schedules URLs, such as Queue, PriorityScheduler, and RedisScheduler.

②, Downloader:Responsible for downloading HTML, such as HttpDownloader, browser Downloader (WebDriver), FiddlerDownloader, and local file Downloader.

③, PageProcesser:Responsible for HTML parsing and new compliant URL Parsing

④, Pipeline:Responsible for data storage, such as MySql, MySqlFile, MSSQL, and MongoDb.

Iii. Others

3.1. I plan to design the entire program as follows:

①. Use WPF for Development (mainly because the interface can be cool)

② Currently, the main functions are: data collection, data storage, data filtering, simulated login (including data publishing), verification code identification, and can be put on the cloud if allowed...

③ Easy to use and visualized operations

④ Well, let's start with that. The most important thing is the reality, which can be expanded slowly later.

3.2 project content may include:

① Parse Html using regular expressions and Xpath

② Verification code recognition

③ Data encryption and decryption

④ Database operations

⑤ A Lot Of content. I believe that after this project is completed, I will have a qualitative leap ....