Python, scrapy Deployment for the CentOS environment

Source: Internet
Author: User
Tags bz2 install openssl

1 Operating system

Centos 6.5 64bit or later

2 Software Environment

Tip: (1) What software to use, (2) the Operating Environment (3) development package.

    1. Python:python-2.7.6.tgz
    2. Scrapy:scrapy-1.0.3.tar.gz
    3. Selenium:selenium (2.48.0)
    4. phantomjs:phantomjs-1.9.8-linux-x86_64.tar.bz2
    5. Mysql-python:mysql-python-1.2.5.zip
    6. Mysql:mysql-5.6.27-1.linux_glibc2.5.x86_64.rpm-bundle.tar
3 Installation Step 1. Installing dependent libraries
yum install zlib-devel bzip2-devel ncurses-devel sqlite-devel readline-devel tk-devel mysql-devel gcc libxslt-devel libxml2-devel libffi-devel  openssl-devel python-devel# yum install zlib-devel# yum install bzip2-devel# yum install openssl-devel# yum install ncurses-devel# yum install sqlite-devel# yum install readline-devel# yum install tk-devel# yum install mysql-devel# yum install gcc# yum install make (centos可能已自带,不用安装)

Installation scrapy must be installed:

# yum install libxslt-devel # yum install libxml2-devel # yum install libffi-devel
2. Upgrade Python

(: https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz)
Centos6.5 system comes with python2.6.6 and needs to be upgraded to Python2.7
Installation library files:yum install zlib-devel openssl-devel sqlite-devel
Download: wget https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz
Extract:tar zxvf Python-2.7.6.tgz
Installation:

cd Python-2.7.6 ./configure --prefix=/usr/local/pythonmakemake install

To create a soft link:ln -sf /usr/local/python/bin/python2.7 /usr/bin/python
Fix Yum: Modify the first line of the file/usr/bin/yum (#!/usr/bin/python to #!/usr/bin/python2.6)

3. Installing Setuptools and Pip

An error occurred while installing the Python module using PIP, it is recommended to update Setuptools and PIP to exclude.

: https://pypi.python.org/packages/source/s/setuptools/setuptools-18.4.tar.gz
Install after decompression: Python setup.py install (or pip install--upgrade setuptools)

: https://pypi.python.org/packages/source/p/pip/pip-7.1.2.tar.gz
Install after decompression: Python setup.py install (or pip install--upgrade pip)

4. Installing SCRAPY+SELENIUM+PHANTOMJS

: https://pypi.python.org/packages/source/S/Scrapy/Scrapy-1.0.3.tar.gz
Install after decompression: python setup.py install (can also be pip install scrapy installed with the command)
Note: Using the pip install scrapy installation may be due to network exceptions or downloading other dependent library times errors, you can download the dependent library for installation separately.

SELENIUM+PHANTOMJS on-Demand installation, if you use the Phantomjs class browser to parse JS install:
: https://pypi.python.org/packages/source/s/selenium/selenium-2.48.0.tar.gz
Install after decompression: python setup.py install (can also be pip install selenium installed with the command)

https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-i686.tar.bz2
wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-i686.tar.bz2

tar jxvf phantomjs-1.9.8-linux-i686.tar.bz2mv phantomjs-1.9.8-linux-i686 /usr/local/src/phantomjsln -sf /usr/local/src/phantomjs/bin/phantomjs /usr/local/bin/phantomjschmod a+x phantomjs 
5. Installing Mysql-python

Python uses the MySQL database and needs to install the module.

Https://pypi.python.org/packages/source/M/MySQL-python/MySQL-python-1.2.5.zip

Install after decompression:python setup.py install

pip install mysql-python

6. Install MySQL


Http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.27-1.linux_glibc2.5.x86_64.rpm-bundle.tar
Install using RPM command after decompression

RpmMysql-server-5.6.27-1. LINUX_GLIBC2.5. x86_64. rpmRpmMysql-client-5.6.27-1.LINUX_GLIBC2 .5.x86_64 .rpmrpm  Mysql-devel-5.6.27-1.linux_glibc2.5.x86_ 64.rpmrpm  Mysql-shared-5.6.27-1.linux_glibc2.5.x86_ 64.rpm           
4 Deployment Note points

1. Do not forget to modify environment variables

2. After installing MySQL, we need to modify the default character set to UTF8

Python, scrapy Deployment for the CentOS environment

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.