Python crawler scrapy Framework Installation

Crawler Scrapy frame, not just a command: PIP3 install scrapy, can be done. (Environment Python3)

Scrapy relies on more cubby, at least to rely on the library has twisted, lxml, Pyopenssl. In different platform environments, it is best to ensure that some basic libraries are installed before installation.

Windows platform:

• Installing lxml

The best way to install it is through the wheel file to install, http://www.lfd.uci.edu/~gohlke/pythonlibs/, from the site to find lxml related files. If it is Python3.5 version, WIndows 64-bit system, then find LXML?3.7.2?CP35?CP35M?WIN_AMD64.WHL this file and download it and then install it via PIP.

After downloading, run the following command to install:

command to install Wheel first: (Better already)

PIP3 Install Wheel

Re-execution:

PIP3 Install LXML?3.7.2?CP35?CP35M?WIN_AMD64.WHL

Installation can also find the LXML?3.7.2?CP35?CP35M?WIN_AMD64.WHL download file:

1. In the blank space, SHIFT + right  click 23. Enter PIP3 install LXML?3.7.2?CP35?CP35M?WIN_AMD64.WHL (or enter PIP3 install LX, to a TAB key, auto-complete)

• Installing Zope.interface

To the official website https://pypi.python.org/pypi/zope.interface#downloads download the corresponding version of the wheel file, and then Pip installs. For example, Python version 3.5, Windows 64-bit system, download ZOPE.INTERFACE-4.3.3-CP35-CP35M-WIN_AMD64.WHL.

You can then install the wheel file with the following command:

PIP3 Install ZOPE.INTERFACE-4.3.3-CP35-CP35M-WIN_AMD64.WHL

• Installing PYOPENSSL

Official website Download wheel file, Https://pypi.python.org/pypi/pyOpenSSL#downloads, If the current version name is PYOPENSSL-16.2.0-PY2.PY3-NONE-ANY.WHL, install it after download.

PIP3 Install PYOPENSSL-16.2.0-PY2.PY3-NONE-ANY.WHL

• Installing twisted

Similarly, http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted下载wheel文件, with PIP installation.
such as Python version 3.5, Windows 64-bit system, download
TWISTED?17.1.0?CP35?CP35M?WIN_AMD64.WHL, and then Pip installs.

PIP3 Install TWISTED?17.1.0?CP35?CP35M?WIN_AMD64.WHL

• Installing Pywin32

Download the corresponding version of the installation package from the official website https://sourceforge.net/projects/pywin32/files/pywin32/Build%20220/.

Downloaded: Pywin32-220.win-amd64-py3.6.exe, Note: The file format is:. exe

1. Double-click File 2. Keep Down (note: There will be a selection of installation paths: D:\python3\)

• Installing Scrapy

Finally install the scrapy, the command is as follows:

PIP3 Install Scrapy

Sometimes, an error occurs when you execute the install scripy directive:

Collecting Scrapyretrying (Retry ( total=4, Connect=none, Read=none, redirect=None)) After connection broken by'Readtimeouterror ("Httpsconnectionpool (host='pypi.python.org', POrt=443): Read timed out. (Read timeout=15)",) ':/simple/qrcode/Retrying (Retry (total=3, Connect=none, Read=none, redirect=None)) After connection broken by'Readtimeouterror ("Httpsconnectionpool (host='pypi.python.org', POrt=443): Read timed out. (Read timeout=15)",) ':/simple/qrcode/

Workaround:

-I https://pypi.douban.com/simplemethod Two: Modify the default software source: Modify The/etc/pip.conf file to be configured for all users. Modify The ~/.pip/pip.conf for the current user configuration. The PIP profile under Windows is C:\Users\xx\pip\pip.ini, and no new is created. [Global]index-url = http://pypi.douban.com/simpletrusted-host = pypi.douban.com

Only one method was used:

Python crawler scrapy Framework Installation