Install scrapy1.0.4 on Centos6.5

Install scrapy1.0.4 on Centos6.5

Recently I learned the crawler framework scrapy by myself. I looked at the official website documents and couldn't wait to install it. The results were terrible. I searched an article on the Internet ).

First, introduce the following environment:

1. Centos6.5 x64 is installed in the vmwarevm. because it needs to be transplanted later, it is compatible with vm10.0.

# Yum-y update

Upgrade System

2. Python2.7.9. The system comes with 2.6.6.

Cd ~ /Download

Pipe Network download source code:

Wget -- no-check-certificatehttps: // www.python.org/ftp/python/2.7.9/Python-2.7.9.tar.xz

Tar xvf Python-2.7.9.tar.xz

Cd Python-2.7.9

./Configure -- with-ensurepip = install # Keep the rest by default

Make

Sudo make install # If python2.7.9 is used as the secondary version, use make altinstall

[Mikky @ localhost Python-2.7.9] $ python
Python 2.7.9 (default, Feb 1 2016, 21:30:54)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

After libffi-devel is upgraded, because there are too many pitfalls on it, it is directly updated. In this step, select:

Sudo yum install update libffi-devel

Now the environment is ready, and the following is the pitfall process o

Installation failure method:

Sudo/usr/local/bin/pip install scrapy

There are many errors. The link at the beginning of the reference document cannot be resolved. You can skip this step.

The main cause of these pitfalls is the dependent libraries of pip software installation, especially cryptography.

By default, pip installs the latest version instead of the most stable version. The latest version of cryptography is installed (1.2.2 my installation time is early February 2016). Therefore:

Sudo/usr/local/bin/pip install cryptography = 0.9

...... Ignore warn

Successfully installedCryptography idna pyasn1 six enum34 ipaddress cffi pycparser
Cleaning up...

By now, we have achieved more than half of the results, and we will invite the main character to play here:

Sudo/usr/local/bin/pip install scrapy
.........

Successfully installed scrapypyOpenSSL queuelib service-identity lxml w3lib cssselect Twisted pyasn1-modules characteristic zope. interface
Cleaning up...

[Mikky @ localhost ~] $ Scrapy version
/Usr/local/lib/python2.7/site-packages/cffi/model. py: 526: UserWarning: 'point _ conversion_form_t 'has no values explicitly defined; next version will refuse to guesswhich integer type it is meant to be (unsigned/signed, int/long)
% Self. _ get_c_name ())
Scrapy 1.0.4
Continue to ignore the warnings. These warnings have gone crazy.

$ Scrapy startproject tutorial
/Usr/local/lib/python2.7/site-packages/cffi/model. py: 526: UserWarning: 'point _ conversion_form_t 'has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
% Self. _ get_c_name ())
New Scrapy project 'tutorial 'created in:
/Home/Mikky/scrapy/tutorial

You can start your first spider:
Cd tutorial
Scrapy genspider example example.com
[Mikky @ localhost scrapy] $ ll
Total 4
Drwxrwxr-x. 3 Mikky 4096 Feb 3 tutorial

Installation Complete, for this warning, see https://bitbucket.org/cffi/cffi/issues/234/cffi-131-userwarning-about.