Python, scrapy Deployment for the CentOS environment

1 Operating system

Centos 6.5 64bit or later

2 Software Environment

Tip: (1) What software to use, (2) the Operating Environment (3) development package.

1. Python:python-2.7.6.tgz
2. Scrapy:scrapy-1.0.3.tar.gz
3. Selenium:selenium (2.48.0)
4. phantomjs:phantomjs-1.9.8-linux-x86_64.tar.bz2
5. Mysql-python:mysql-python-1.2.5.zip
6. Mysql:mysql-5.6.27-1.linux_glibc2.5.x86_64.rpm-bundle.tar

3 Installation Step 1. Installing dependent libraries

yum install zlib-devel bzip2-devel ncurses-devel sqlite-devel readline-devel tk-devel mysql-devel gcc libxslt-devel libxml2-devel libffi-devel  openssl-devel python-devel# yum install zlib-devel# yum install bzip2-devel# yum install openssl-devel# yum install ncurses-devel# yum install sqlite-devel# yum install readline-devel# yum install tk-devel# yum install mysql-devel# yum install gcc# yum install make (centos可能已自带，不用安装)

Installation scrapy must be installed:

# yum install libxslt-devel # yum install libxml2-devel # yum install libffi-devel

2. Upgrade Python

(: https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz)
Centos6.5 system comes with python2.6.6 and needs to be upgraded to Python2.7
Installation library files:yum install zlib-devel openssl-devel sqlite-devel
Download: wget https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz
Extract:tar zxvf Python-2.7.6.tgz
Installation:

cd Python-2.7.6 ./configure --prefix=/usr/local/pythonmakemake install

To create a soft link:ln -sf /usr/local/python/bin/python2.7 /usr/bin/python
Fix Yum: Modify the first line of the file/usr/bin/yum (#!/usr/bin/python to #!/usr/bin/python2.6)

3. Installing Setuptools and Pip

An error occurred while installing the Python module using PIP, it is recommended to update Setuptools and PIP to exclude.

: https://pypi.python.org/packages/source/s/setuptools/setuptools-18.4.tar.gz
Install after decompression: Python setup.py install (or pip install--upgrade setuptools)

: https://pypi.python.org/packages/source/p/pip/pip-7.1.2.tar.gz
Install after decompression: Python setup.py install (or pip install--upgrade pip)

4. Installing SCRAPY+SELENIUM+PHANTOMJS

: https://pypi.python.org/packages/source/S/Scrapy/Scrapy-1.0.3.tar.gz
Install after decompression: python setup.py install (can also be pip install scrapy installed with the command)
Note: Using the pip install scrapy installation may be due to network exceptions or downloading other dependent library times errors, you can download the dependent library for installation separately.

SELENIUM+PHANTOMJS on-Demand installation, if you use the Phantomjs class browser to parse JS install:
: https://pypi.python.org/packages/source/s/selenium/selenium-2.48.0.tar.gz
Install after decompression: python setup.py install (can also be pip install selenium installed with the command)
：
https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-i686.tar.bz2
wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-i686.tar.bz2

tar jxvf phantomjs-1.9.8-linux-i686.tar.bz2mv phantomjs-1.9.8-linux-i686 /usr/local/src/phantomjsln -sf /usr/local/src/phantomjs/bin/phantomjs /usr/local/bin/phantomjschmod a+x phantomjs 

5. Installing Mysql-python

Python uses the MySQL database and needs to install the module.
：
Https://pypi.python.org/packages/source/M/MySQL-python/MySQL-python-1.2.5.zip

Install after decompression:python setup.py install

pip install mysql-python

6. Install MySQL

：
Http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.27-1.linux_glibc2.5.x86_64.rpm-bundle.tar
Install using RPM command after decompression

RpmMysql-server-5.6.27-1. LINUX_GLIBC2.5. x86_64. rpmRpmMysql-client-5.6.27-1.LINUX_GLIBC2 .5.x86_64 .rpmrpm  Mysql-devel-5.6.27-1.linux_glibc2.5.x86_ 64.rpmrpm  Mysql-shared-5.6.27-1.linux_glibc2.5.x86_ 64.rpm           

4 Deployment Note points

1. Do not forget to modify environment variables

2. After installing MySQL, we need to modify the default character set to UTF8

Python, scrapy Deployment for the CentOS environment