Python crawler scrapy Framework Primer (0)

Want to learn the crawler, but also want to understand the Python language, there is a python expert recommend me to see Scrapy.

Scrapy is a Python crawler framework, which is said to be flexible, and there is a lot of information on the web about the framework, which is not covered here. Keep track of the problems and solutions that I have encountered.

For a few links, I'm trying to learn from these things:

Scrapy Chinese document (0.24 version, I study when Scrapy already 1.1, perhaps some outdated):

Http://scrapy-chs.readthedocs.io/zh_CN/0.24/intro/overview.html

Great God's Blog introduction:

Getting Started example: http://www.cnblogs.com/txw1958/archive/2012/07/16/scrapy-tutorial.html

Installation: http://www.cnblogs.com/txw1958/archive/2012/07/12/scrapy_installation_introduce.html

Scrapy Official website (don't expect to download it on this, anyway I didn't download scrapy):

http://scrapy.org/

Not recommended to download on the official website!!!

Why not recommend downloading on the official website (I did not find the official website to download the WHL address of the current version of Scrapy, which may be available for download), because scrapy needs a lot of things to rely on. According to the Great God's blog, if you download it manually, you must install at least these several dependent packages:

1. python--Basic Language Platform
2. twisted--Event-driven network engine
3. W3lib, libxml2--a library file
4. PYOPENSSL--SSL protocol, for remote control and the like, I guess it might have something to do with the distributed crawler.

The scrapy is then installed. It's a lot of trouble. So it is recommended to install directly with PIP (Scrapy is also recommended PIP installation). After the native PIP is installed, you only need to enter it at the command line:

Pip Install Scrapy

A command will be OK. Simple and convenient. However, PIP installation itself will encounter a lot of problems, and also open the article elaborated.

This scrapy installation is complete. Anyway, as Python and scrapy double small white I, in the setting up the environment when encountered a lot of trouble. In the attitude of learning and record, I will put the difficulties and solutions to record. The article will also change slowly later. If someone really read this blog post, please tap the spray, thank you ~ ╮(╯▽╰)╭

Python crawler scrapy Framework Primer (0)