Install Python, scrapy, Redis, MySQL under Linux

Today to the online server to install the crawler environment, casually record the installation process, there are many similar installation process, I just collation + verification, I hope to be installed to help people

Install Python installation python

wget https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tgztar zxvf Python-2.7.11.tgz cd Python-2.7.11./configure --prefix=/usr/localmake && make altinstall

Check Python version
Python-v

Installing Scrapy Installation Python-devel

Yum Search All Python-devel
Yum Install python-devel.x86_64

Installing Setuptools

wget --no-check-certificatehttps://bootstrap.pypa.io/ez_setup.py

–2016-05-10 10:48:35–https://bootstrap.pypa.io/ez_setup.py
Resolving Bootstrap.pypa.io ... 103.245.222.175
Connecting to bootstrap.pypa.io|103.245.222.175|:443 ... connected.
Warning:certificate common name "*.c.ssl.fastly.net" doesn ' t match requested host name "Bootstrap.pypa.io".
HTTP request sent, awaiting response ... OK
length:12385 (12K) [Text/x-python]
Saving to: "ez_setup.py"

100%[========================================================================================================== ] 12,385–.-k/s in 0s

2016-05-10 10:48:35 (+ MB/s)-"ez_setup.py" saved [12385/12385]

ll

Total 83956
-rw-r–r–1 root root 12385 May 6 21:30 ez_setup.py

pythonez_setup.py--insecure

Downloading Https://pypi.io/packages/source/s/setuptools/setuptools-21.0.0.zip
Extracting IN/TMP/TMP9OQVDP
Now working in/tmp/tmp9oqvdp/setuptools-21.0.0
Installing Setuptools
Running Install
Running Bdist_egg
Running Egg_info
Writing Requirements to Setuptools.egg-info/requires.txt
................
Copying Setuptools-21.0.0-py2.7.egg To/usr/local/lib/python2.7/site-packages
Adding setuptools 21.0.0 to easy-install.pth file
Installing Easy_install Script To/usr/local/bin
Installing easy_install-2.7 Script To/usr/local/bin

Installed/usr/local/lib/python2.7/site-packages/setuptools-21.0.0-py2.7.egg
Processing dependencies for setuptools==21.0.0
Finished processing dependencies for setuptools==21.0.0

PIP installation

1. Download file
   Note: If the first execution is not downloaded, you can perform a few more wget

https://bootstrap.pypa.io/get--no-check-certificate

1. Performing the installation

get-pip.py

1. View version

pip –V

Zope.interface Installation

Web address: Https://pypi.python.org/pypi/zope.interface/4.1.3#downloads

tar -zxvf zope.interface-4.1.3.tar.gz zope.interface-4.1.3-py2.7-linux-x86_64.eggcd zope.interface-4.1.3python setup.py buildpython setup.py install

After the installation is complete, you can see the following/usr/local/lib/python2.7/site-packages
Zope.interface-4.1.3-py2.7-linux-x86_64.egg

Installing OpenSSL

wget https://www.openssl.org/source/openssl-1.0.2h.tar.gztar -zxvf openssl-1.0.2h.tar.gz cd openssl-1.0.2h./configmakemake testmake install

View the version of OpenSSL
OpenSSL version
If it is an older version, execute the following command:

mv /usr/bin/openssl /root/ln -s /usr/local/ssl/bin/openssl /usr/bin/openssl

View version again

version

OpenSSL 1.0.2h 3 May 2016

PYOPENSSL Installation

yum install gcc libffi-devel python-devel openssl-devel   yum install openssl-devel python-devel python-sphinx

：
Https://pypi.python.org/pypi/pyOpenSSL/0.15.1/pyOpenSSL-0.15.1.tar.gz

If not successful on Linux, perform more easy_install several times

easy_install https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.15.1.tar.gz#md5=f447644afcbd5f0a1f47350fec63a4c6

Twisted Installation

wget http://twistedmatrix.com/Releases/Twisted/12.0/Twisted-12.0.0.tar.bz2bzip2 -d Twisted-12.0.0.tar.bz2tar -xvf Twisted-12.0.0.tarcd Twisted-12.0.0python setup.py install

W3lib Installation

wget--no-check-certificatehttp://pypi.python.org/packages/source/w/w3lib/w3lib-1.0. tar.gzwgethttp://pypi.python.org/packages/source/w/w3lib/w3lib-1.0. tar.gz[root@localhost/opt/w3lib-1.0]$ Tar-xvzf w3lib-1.0. tar.gz [Root@localhost/opt/w3lib-1.0]$ CD w3lib-1.0[Root@localhost/opt/w3lib-1.0]$ Python setup.py Install

After the installation is complete, you can see W3lib and w3lib-1.0-py2.7.egg-info below the/usr/local/lib/python2.7/site-packages.

Installing LIBXML2

As shown below:
Http://download.chinaunix.net/down.php?id=28497&ResourceID=6095&site=1
Libxml2-2.7.4.tar.gz

cd libxml2-2.7.4./configuremakemake install需要pythonpythonpython setup.py install

Note:
Compile libxml2 times wrong,./.libs/libxml2.so:undefined reference to ' Gzopen64 ':
1, download zlib-1.2.3.tar.gz, unzip, configure, VI Makefile, then add-fpic parameters after cflags, compile make and make install
2, recompile libxml2,
./configure–prefix=/usr/local/libxml2–with-zlib=/usr/local/zlib/
Then VI Makefile, then add-fpic parameters after cflags, compile make and make install
Ok

lxml Installation

all libxslt-develyum install libxslt-devel.x86_64pip install lxml

Installing Scrapy

wget --no-check-certificate https://pypi.python.org/packages/source/S/Scrapy/Scrapy-1.0.3.tar.gz#md5=cefb981076a26112d6f9b76cb7e5ba7d tar -xvzf  Scrapy-1.0.3.tar.gz  cd Scrapy-1.0.3  python setup.py install  

Installing Redis

1. Install the required support environment

The first thing to do before you install Redis is the TCL tool that installs Unix, and you won't be able to test Redis later if you don't install it. The following error message is returned when you execute make test later: Need TCL 8.xuyao de5 or newer in order to run the Redis test, the specific process is:
Copy the code code as follows:

cd /usr/local/srcwget http://downloads.sourceforge.net/tcl/tcl8.6.3-src.tar.gztar -zxvf tcl8.6.3-src.tar.gzcd ?tcl8.6.3/unix/./configuremakemake install

1. Installing Redis

The process of installing Redis is very simple and the specific tutorials are available on the website. Specific as follows: Http://redis.io/download
Copy the code code as follows:

cd /usr/local/src$ wget http://download.redis.io/releases/redis-3.0.5.tar.gztar zxvf redis-3.0.5.tar.gzcd redis-3.0.5makemake PREFIX=/usr/local/redis install

Where Prefix=/usr/local/redis can be omitted and, in the case of ellipsis, Redis is installed by default into the/usr/local/bin directory.
3. Test Redis

The code is as follows:

cd srcmake test

With the above command, Redis can be tested in greater measure.

1. To view the installation version:

redis-server--version

MySQL installation install MySQL database

Http://www.cnblogs.com/xiaoluo501395377/archive/2013/04/07/3003278.html

Installing Mysql-python

pip install mysql-python验证：>>> import MySQLdb>>> 

Success!

Install Python, scrapy, Redis, MySQL under Linux