Install scrapy1.0.4 on Centos6.5

Recently I learned the crawler framework scrapy by myself. I looked at the official website documents and couldn't wait to install it. The results were terrible. I searched an article on the Internet ). First, introduce the environment: 1. Centos6.5x64 is installed in the vmwarevm. because it needs to be transplanted later, it is compatible with vm10.0. # Yum-yupda

Recently I learned the crawler framework scrapy by myself. I looked at the official website documents and couldn't wait to install it. The results were terrible. I searched an article on the Internet ).

First, introduce the following environment:

1. Centos6.5 x64 is installed in the vmwarevm. because it needs to be transplanted later, it is compatible with vm10.0.

# Yum-y update

Upgrade system

2. Python2.7.9. The system comes with 2.6.6.

Cd ~ /Download

Pipe network download source code:

Wget -- no-check-certificate https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tar.xz

Tar xvf Python-2.7.9.tar.xz

Cd Python-2.7.9

./Configure -- with-ensurepip = install # Keep the rest by default

Make

Sudo make install # if python2.7.9 is used as the secondary version, use make altinstall

[Mikky @ localhost Python-2.7.9] $ python
Python 2.7.9 (default, Feb 1 2016, 21:30:54)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

After libffi-devel is upgraded, because there are too many pitfalls on it, it is directly Updated. in this step, select:

Sudo yum install update libffi-devel
 

Now the environment is ready. The following is the step o (? □ ?) O

Installation failure method:

Sudo/usr/local/bin/pip install scrapy

There are many errors. the link at the beginning of the reference document cannot be resolved. You can skip this step.

The main cause of these pitfalls is the dependent libraries of pip software installation, especially cryptography.

By default, pip installs the latest version instead of the most stable version. The latest version of cryptography is installed (1.2.2 my installation time is early February 2016). therefore:

Sudo/usr/local/bin/pip install cryptography = 0.9

...... Ignore warn

Successfully installedCryptography idna pyasn1 six enum34 ipaddress cffi pycparser
Cleaning up...

By now, we have achieved more than half of the results, and we will invite the main character to play here:

Sudo/usr/local/bin/pip install scrapy
.........

Successfully installed scrapyPyOpenSSL queuelib service-identity lxml w3lib CSS select Twisted pyasn1-modules characteristic zope. interface
Cleaning up...

[Mikky @ localhost ~] $ Scrapy version
/Usr/local/lib/python2.7/site-packages/cffi/model. py: 526: UserWarning: 'Point _ conversion_form_t 'has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
% Self. _ get_c_name ())
Scrapy 1.0.4
Continue to ignore the warnings. these warnings have gone crazy.

$ Scrapy startproject tutorial
/Usr/local/lib/python2.7/site-packages/cffi/model. py: 526: UserWarning: 'Point _ conversion_form_t 'has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
% Self. _ get_c_name ())
New Scrapy project 'tutorial 'created in:
/Home/Mikky/scrapy/tutorial

You can start your first spider:
Cd tutorial
Scrapy genspider example example.com
[Mikky @ localhost scrapy] $ ll
Total 4
Drwxrwxr-x. 3 Mikky 4096 Feb 3 tutorial

 

Installation complete, for this warning, see https://bitbucket.org/cffi/cffi/issues/234/cffi-131-userwarning-about.