Install scrapy and scrapyd, and install scrapyscrapyd.

Install scrapy and scrapyd, and install scrapyscrapyd.

Scrapy is a crawler program written in python.

Scrapyd is an application for deploying and running scrapy crawlers. It provides json api calling methods to deploy and control crawlers.

This article verifies that fedora and centos are successfully installed.

Run the following program:

L python2.7

L pip setuptools

L lxml

L OpenSSL (pyopenssl)

Scrapyd is required.6800Port

Install dependency:

Yum-y install readline-devel

Yum-y install openssl-devel

Yum-y install sqlite-devel

InstallPython2.7

Wget https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tgz

Tar zxvf Python-2.7.9.tgz

Cd Python-2.7.9

./Configure -- with-zlib-dir =/usr/local/lib

Make & make install

InstallPip:

Wget https://bootstrap.pypa.io/get-pip.py

Python2.7 get-pip.py

Setuptools will also be installed.

InstallLxmlAnd Dependencies

Yum install libffi-devel

Yum install libxml2

Yum install libxslt

Yum install libxml2-devel

Yum install libxslt-devel

Pip install lxml

InstallPyOpenSSL

Https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.15.1.tar.gz#md5=f447644afcbd5f0a1f47350fec63a4c6 -- no-check-certificate

Tar zxvf pyOpenSSL-0.15.1.tar

Cd pyOpenSSL-0.15.1

Python2.7 setup. py install

InstallTwisted

Yum-y install bzip2-devel

Wget https://pypi.python.org/packages/source/T/Twisted/Twisted-14.0.0.tar.bz2#md5=9625c094e0a18da77faa4627b98c9815

Tar xf Twisted-14.0.0.tar.bz2

CDS Twisted-14.0.0

Python2.7 setup. py install

InstallScrapy:

Pip install Scrapy

Test and install:

Scrappy startproject testProject. If the path in scrapy format can be generated in the current directory, the installation is successful.

InstallScrapyd

Pip install scrapyd

InstallScrapyd-client

Wget https://github.com/scrapy/scrapyd-client/archive/master.zip

Unzip master.zip

Cd scrapyd-client-master

Python2.7 setup. py install

Scrapyd-client is used to deploy the scrapy program: scrapyd-deploy on the server.