Python Crawler's scrapy installation

Summary: Scrapy,python develops a fast, high-level screen capture and Web Capture framework for crawling Web sites and extracting structured data from pages.　　Scrapy can be used for data mining, monitoring and automated testing in a wide range of applications. The attraction of Scrapy is that it is a framework that anyone can easily modify as needed. It also provides a variety of types of crawler base classes, such as Basespider, Sitemap Crawler, and the latest version of the web2.0 crawler to provide support.

Installation Environment:

Windows installation:

　　Attention:

1. Install the dependent package from the top down, then the. whl file directly pip3 the install file absolute path and name

2, pay attention to your PIP version, download more than 9.0 version.

　　Linux:

The following is an introduction to the Centos6.5 version installation, note the configuration of the Yum source.

Install the dependent packages.

Yum install Python3-devyum install-y python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev Libssl-dev

Check that the PIP version is not 9.0.1

[[email protected]/]# PIP3--versionpip 9.0.1 from/usr/local/lib/python3.6/site-packages (Python 3.6)

　　

Start installing Scrapy. (The Linux system does not depend on the package like Windows requires one of us to install, it will automatically install all the dependencies required to install, omit a lot of steps and problems)

[Email protected]/]#PIP3 Install scrapyCollecting scrapy downloading SCRAPY-1.4.0-PY2.PY3-NONE-ANY.WHL (248kB) 100% |████████████████████████████████| 256kB 29kb/scollecting pydispatcher>=2.0.5 (from scrapy) downloading pydispatcher-2.0.5.tar.gzcollecting Parsel  >=1.1 (from scrapy) downloading Parsel-1.2.0-py2.py3-none-any.whlcollecting service-identity (from Scrapy) Downloading Service_identity-17.0.0-py2.py3-none-any.whlcollecting w3lib>=1.17.0 (from Scrapy) Downloading W3lib-1.18.0-py2.py3-none-any.whlcollecting Queuelib (from scrapy) downloading Queuelib-1.4.2-py2.py3-none-any.whlcollecting lxml (from scrapy) downloading lxml-4.1.0-cp36-cp36m-manylinux1_x86_ 64.WHL (5.6MB) 100% |████████████████████████████████| 5.6MB 13kb/scollecting cssselect>=0.9 (from scrapy) downloading cssselect-1.0.1-py2.py3-none-any.whlcollecting six >=1.5.2 (from scrapy) downloading Six-1.11.0-py2.py3-none-any.whlcollecting pyopenssl (from scrapy) Downloading pyOp ENSSL-17.3.0-PY2.PY3-NONE-ANY.WHL (51kB) 100% |████████████████████████████████| 51kB 11kb/scollecting twisted>=13.1.0 (from scrapy) downloading twisted-17.9.0.tar.bz2 (3.0MB) 100% |██████████████ ██████████████████| 3.0MB 20kb/scollecting attrs (from service-identity->scrapy) downloading Attrs-17.2.0-py2.py3-none-any.whlcollecting pyasn1-modules (from service-identity->scrapy) Downloading pyasn1_ MODULES-0.1.5-PY2.PY3-NONE-ANY.WHL (60kB) 100% |████████████████████████████████| 61kB 79kb/scollecting pyasn1 (from service-identity->scrapy) downloading PYASN1-0.3.7-PY2.PY3-NONE-ANY.WHL (63kB) 1 0% |████████████████████████████████| 71kB 87kb/scollecting cryptography>=1.9 (from pyopenssl->scrapy) downloading CRYPTOGRAPHY-2.1.2-CP36-CP36M-MANYLINUX1_X86_64.WHL (2.2MB) 100% |████████████████████████████████| 2.2MB 16kb/scollecting zope.interface>=4.0.2 (from twisted>=13.1.0->scrapy) downloading ZOPE.INTERFACE-4.4.3-CP36-CP36M-MANYLINUX1_X86_64.WHL (173kB) 100% |████████████████████████████████| 174kB 20kb/scollecting constantly>=15.1 (from twisted>=13.1.0->scrapy) downloading  Constantly-15.1.0-py2.py3-none-any.whlcollecting incremental>=16.10.1 (from Twisted>=13.1.0->scrapy) Downloading Incremental-17.5.0-py2.py3-none-any.whlcollecting automat>=0.3.0 (from Twisted>=13.1.0-> scrapy) downloading Automat-0.6.0-py2.py3-none-any.whlcollecting hyperlink>=17.1.1 (from Twisted>=13.1.0-> scrapy) downloading HYPERLINK-17.3.1-PY2.PY3-NONE-ANY.WHL (73kB) 100% |████████████████████████████████| 81kB 46kb/scollecting idna>=2.1 (from cryptography>=1.9->pyopenssl->scrapy) downloading IDNA-2.6-PY2.PY3-NONE-ANY.WHL (56kB) 100% |████████████████████████████████| 61kB 48kb/scollecting asn1crypto>=0.21.0 (from Cryptography>=1.9->pyopenssl->scrapy) Downloading ASN1CRYPTO-0.23.0-PY2.PY3-NONE-ANY.WHL (99kB) 100% |████████████████████████████████| 102kB 19kb/scollecting cffi>=1.7; Platform_python_implementation! = "PyPy" (from cryptography>=1.9->pyopenssl->scrapy) downloading CFFI-1.11.2-CP36-CP36M-MANYLINUX1_X86_64.WHL (419kB) 100% |████████████████████████████████| 430kB 18kb/srequirement already Satisfied:setuptools in/usr/local/lib/python3.6/site-packages (from Zope.interface >=4.0.2->twisted>=13.1.0->scrapy) Collecting Pycparser (from cffi>=1.7; platform_python_ Implementation! = "PyPy"->cryptography>=1.9->pyopenssl->scrapy) downloading pycparser-2.18.tar.gz ( 245kB) 100% |████████████████████████████████|  256kB 62kb/sinstalling collected Packages:pydispatcher, six, W3lib, lxml, Cssselect, Parsel, Idna, Asn1crypto, Pycparser, Cffi, cryptography, Pyopenssl, Attrs, PYASN1, Pyasn1-modules, service-identity, Queuelib, Zope.interface, constantly, in Cremental, Automat, Hyperlink, Twisted, scrapy Running setup.py install for pydispatcher ... done Running setup.py Insta ll for Pycparser ... do Running setup.py install for Twisted ... donesuccessfully installed Automat-0.6.0 PyDispatcher-2.0.5 Twisted-17.9.0 asn1crypto-0.23.0 attrs-17.2.0 cffi-1.11.2 constantly-15.1.0 cryptography-2.1.2 cssselect-1.0.1 hyperlink-17.3.1 idna-2.6 incremental-17.5.0 lxml-4.1.0 parsel-1.2.0 pyOpenSSL-17.3.0 pyasn1-0.3.7 pyasn1-modules-0.1.5 pycparser-2.18 queuelib-1.4.2 scrapy-1.4.0  service-identity-17.0.0 six-1.11.0 w3lib-1.18.0 zope.interface-4.4.3[[email protected]/]#Python3Python 3.6.3 (default, OCT, 10:18:57) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on Linuxtype "help", "copyright", "cre Dits "or" license "for more information.>>>Import Scrapy>>>

　　

Python Crawler's scrapy installation