Python crawler--scrapy Framework Installation

When writing a Python crawler, we can do most of the requirements with libraries such as requests and selenium, but when the amount of data is too large or there is a certain requirement for crawl speed, the advantage of using the framework to write is reflected. With the help of the framework, not only the program architecture will be much clearer, but also the crawl efficiency will increase, so the crawler framework is a good way to write a crawler.

For the Python crawler framework, Scrapy is currently hot, which is an application framework for crawling web-structured data. Scrapy is a powerful framework, depending on the library is also more, such as the Lxml,pyopenssl and twisted, and so on, these libraries in different platform requirements are not the same, though, but in fact the installation is relatively simple, but sometimes the installation process error but a lot, The installation method and error handling methods are described below.

Author Environment: Win10 + python3.6.5 Installation method

Although it is mentioned earlier that Scrapy relies on more libraries, it can be installed using the PIP command directly, and some dependent libraries will be installed automatically:

Pip Install Scrapy

Verifying the installation

After installation, enter scrapy directly at the command line, indicating that the installation was successful if the output resembles the following

Error handling(section, which lists errors encountered during the author's installation)①Error:microsoft Visual C + + 14.0 is required

This error is obvious, follow the prompts to install Microsoft Visual C + + on the line, I later download and install Visual Studio 2017 on the official website to resolve this issue

Iiunicodedecodeerror: ' utf-8 ' codec can ' t decode byte 0xce in posiyion 98:invalid continuation byte

This error is a coding problem, at first I was also a blank face, after consulting the data found a lot of solutions, in fact, the reason for the error is very simple, because the encoding format in Windows is ' GBK ', so the installation of ' Utf-8 ' does not (the error message also shows that can not encode), then how to solve it?

First, depending on the error stream, determine the error file location: C:\Users\Littl\AppData\Local\Programs\Python\Python36\Lib\site-packages\pip\compat, open File __init __.py, find near line 73, change ' utf-8 ' all to ' gb2312 ', save install again

The above for the author installation process encountered problems, in the row wrong also found in fact for a lot of unknown Li Error, slowly understand the error hints and flow can be a good solution, if there are other problems can be based on error tips to check the relevant online information, also welcome message learning Exchange.

Python crawler--scrapy Framework Installation