Installation
Linux and Mac direct pip install Scrapy on the line
Windows installation Steps
a. pip3 install wheel b. 下载twisted http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted c. 进入下载目录，执行 pip3 install Twisted?17.1.0?cp35?cp35m?win_amd64.whl d. pip3 install scrapy e. 下载并安装pywin32：https://sourceforge.net/projects/pywin32/files/Scrapy
Scrapy is an application framework written to crawl Web site data and extract structural data. It can be used in a series of programs such as data mining, information processing or storing historical data.
It was originally designed for page fetching (more specifically, network crawling) and could also be applied to get the data returned by the API (for example, Amazon Associates Web Services) or a generic web crawler. Scrapy can be used for data mining, monitoring and automated testing in a wide range of applications.
Scrapy uses the Twisted asynchronous network library to handle network traffic. The overall structure is broadly as follows
Scrapy mainly includes the following components:
- Engine (Scrapy)
Data flow processing for the entire system, triggering transactions (framework core)
- Scheduler (Scheduler)
Used to accept requests sent by the engine, pressed into the queue, and returned when the engine was requested again. It can be imagined as a priority queue for a URL (crawling the URL of a Web page or a link), which determines what the next URL to crawl is, and removes duplicate URLs
- Downloader (Downloader)
Used to download Web content and return Web content to spiders (Scrapy downloader is built on twisted, an efficient asynchronous model)
- Reptile (Spiders)
Crawlers are primarily working to extract the information they need from a particular Web page, the so-called entity (Item). The user can also extract a link from it, allowing Scrapy to continue scratching a page
- Project Pipeline (Pipeline)
Responsible for processing the entities extracted from the Web page, the main function is to persist the entity, verify the validity of the entity, eliminate unnecessary information. When the page is parsed by the crawler, it is sent to the project pipeline, and the data is processed in several specific order.
- Downloader middleware (Downloader middlewares)
The framework is located between the Scrapy engine and the downloader, primarily to handle requests and responses between the Scrapy engine and the downloader.
- Crawler middleware (spider middlewares)
Between the Scrapy engine and the crawler, the main task is to handle the spider's response input and request output.
- Dispatch middleware (Scheduler middewares)
The middleware between the Scrapy engine and the dispatch, sent from the Scrapy engine to the scheduled request and response.
The scrapy running process is probably as follows:
- The engine pulls a link (URL) from the scheduler for the next crawl
- The engine encapsulates the URL into a request to pass to the downloader
- The Downloader downloads the resource and encapsulates it as a response packet (Response)
- Reptile parsing response
- Resolves the entity (Item) and gives it to the entity pipeline for further processing
- Parse out a link (URL), then the URL to the scheduler waiting to crawl
1. Basic commands
| 1234567891011121314151617 |
1.scrapy startproject 项目名称 -在当前目录中创建中创建一个项目文件（类似于Django）2.scrapy genspider [-t template] <name> <domain> -创建爬虫应用 如： scrapy gensipider -t basic oldboy oldboy.com scrapy gensipider -t xmlfeed autohome autohome.com.cn PS: 查看所有命令：scrapy gensipider -l 查看模板命令：scrapy gensipider -d 模板名称3.scrapy list -展示爬虫应用列表4.scrapy crawl 爬虫应用名称 -运行单独爬虫应用 |
Scrapy Installation and process
