Scrapy Work Flow

Scrapy mainly has the following components:

1, Engine (scrapy)

Used to process the entire system's data flow, triggering transactions (framework core)

2, Scheduler (Scheduler)

Used to receive a request from the engine, pressed into the queue, and returned when the engine requests again, can be imagined as a URL (crawl web site URL or link) Priority queue, it determines the next crawl URL is what, while removing duplicate URLs

3, Downloader (Downloader)

To download the content of the Web page and return the contents of the Web page to the spider (Scrapy downloader is built again twisted) on this efficient asynchronous model

4, Reptile (Spiders)

Crawlers are primarily working to extract the information they need from a particular Web page, the so-called entity (Item). The user can also extract a link from it, allowing Scrapy to continue scratching a page

5, Project Pipeline (Pipeline)

Responsible for dealing with the entities extracted from the Web page, the main function is to persist the entity, verify the validity of the entity, and clear the unwanted information. When the page is parsed by the crawler, it is sent to the project pipeline, and the data is processed in several specific order.

6, Downloader middleware (Downloader middlewares)

A framework between the Scrapy engine and the downloader, primarily dealing with requests and responses between the Scarpy engine and the downloader

7, crawler middleware (spider middlewares)

A framework between the Scrapy engine and the crawler, mainly working to handle the spider's response input and request output

8, Dispatch middleware (Scheduler middewares)

A dispatch request and response sent from the Scrapy engine between the middleware between the Scrapy engine and the dispatch.

The scrapy running process is probably as follows:
1, the engine pulls a link (URL) from the scheduler for the next crawl

2, the engine encapsulates the URL as a request to pass to the downloader

3, the Downloader downloads the resource and encapsulates it as a response packet (Response)

4, Reptile parsing response

5, the Entity (ITEM) is parsed, then the entity pipeline is processed further.

6, parse out a link (URL), then the URL to the scheduler waiting to crawl

Scrapy Work Flow