"Python3 crawler" 05_ installation Scrapy

Scrapy Introduction

Scrapy is an application framework written with pure Python for crawling Web site data and extracting structural data, which is very versatile. The power of the framework, users only need to customize the development of a few modules can be easily implemented a crawler, used to crawl Web content and a variety of pictures, very convenient.

• Scrapy Engine(引擎): Responsible Spider ,,, ItemPipeline Downloader Scheduler intermediary communication, signal, data transmission and so on.

• Scheduler(调度器): It is responsible for receiving 引擎 requests sent from the request, and in a certain way to arrange the arrangement, the queue, when 引擎 necessary, returned 引擎 .

• Downloader（下载器）: Responsible for downloading Scrapy Engine(引擎) all requests requests sent, and returning them to the responses to be Scrapy Engine(引擎) 引擎 processed by hand Spider ,

• Spider（爬虫）: It handles all responses, extracts data from it, gets the data needed for the item field, submits the URL that needs to be followed, and 引擎 enters again Scheduler(调度器) ,

• Item Pipeline(管道): It is responsible for processing the Spider item retrieved and for post-processing (detailed analysis, filtering, storage, etc.).

• Downloader Middlewares（下载中间件）: You can be seen as a component that can customize the extended download feature.

• Spider Middlewares（Spider中间件）: You can understand that it is a functional component that can be customized for expansion and operation 引擎 and Spider intermediate 通信 (such as Spider the entry of the responses; and the requests from the Spider exit)

Installing Scrapy

System Configuration WIN10 64-bit Enterprise Edition

Install scrapy directly using the command

Pip Install Scrapy

Runtimeerror:cryptography requires Setuptools 18.5 or newer, please upgrade to a newer version of Setuptools

Problem solving:

Download Setuptools and install

Then run pip install Scrapy again and error again:

Error:microsoft Visual C + + 10.0 is required. Get it with "Microsoft Windows SDK 7.1": www.microsoft.com/download/details.aspx?id=8279

Error:microsoft Visual C + + 10.0 is required.

Solve the problem:

According to the URL provided by the error, click Go to download the file directly and install

"Python3 crawler" 05_ installation Scrapy