"Scrapy" Python develops a fast, high-level screen capture and web crawling framework for crawling web sites and extracting structured data from pages. Scrapy can be used for data mining, monitoring and automated testing in a wide range of applications. The attraction of Scrapy is that it is a framework that anyone can easily modify as needed. It also provides a variety of types of crawler base classes, such as Basespider, Sitemap Crawler, and the latest version of the web2.0 crawler to provide support. The crawler execution Framework Scrapy uses the twisted asynchronous network library to handle network traffic. The overall structure is broadly as follows:
Scrapy mainly includes the following components:
- Engine (Scrapy)
Data flow processing for the entire system, triggering transactions (framework core)
- Scheduler (Scheduler)
used to accept requests sent by the engine, pressed into the queue, and returned when the engine was requested again. You can imagine the priority queue of a URL (the URL that crawls a Web page or a link), which determines what the next URL is to crawl, and removes the duplicate URLs
- Downloader (Downloader)
used to download Web content and return Web content to spiders (Scrapy downloader is built on twisted, an efficient asynchronous model)
- Reptile (Spiders)
crawlers are primarily working to extract the information they need from a particular Web page, the so-called entity (Item). The user can also extract a link from it, allowing Scrapy to continue scratching a page
- Project Pipeline (Pipeline)
responsible for processing the entities extracted from the Web page, the main function is to persist the entity, verify the validity of the entity, eliminate unnecessary information. When the page is parsed by the crawler, it is sent to the project pipeline, and the data is processed in several specific order.
- Downloader middleware (Downloader middlewares)
The framework is located between the Scrapy engine and the downloader, primarily to handle requests and responses between the Scrapy engine and the downloader.
- Crawler middleware (spider middlewares)
between the Scrapy engine and the crawler, the main task is to handle the spider's response input and request output.
- Dispatch middleware (Scheduler middewares)
The middleware between the Scrapy engine and the dispatch, sent from the Scrapy engine to the scheduled request and response.
The scrapy running process is probably as follows:
- The engine pulls a link (URL) from the scheduler for the next crawl
- The engine encapsulates the URL into a request to pass to the downloader
- The Downloader downloads the resource and encapsulates it as a response packet (Response)
- Reptile parsing response
- Resolves the entity (Item) and gives it to the entity pipeline for further processing
- Parse out a link (URL), then the URL to the scheduler waiting to crawl
URL Manager 1, add a new URL to the collection of URLs to crawl. 2. Determine if the URL to be added is in the container (including the collection of URL to crawl and the collection of crawled URLs). 3. Get the URL you want to crawl. 4, determine whether to crawl the URL. 5. Move the crawl completed URL from the collection of crawled URLs to the crawled URL collection. "Storage Options" 1, Memory (python memory) to crawl URL collection: Set () Crawled URL collection: Set () 2, relational database (MySQL) URLs (URL, is_crawled) 3, Cache (
Redis) to crawl the URL collection: Set crawled URL collection: Set large Internet companies, due to the high performance of the cache database, the URL is generally stored in the cache database. Small companies typically store URLs in memory and store them in a relational database if they want to store them permanently.
"Python" crawler-scrapy
