Deep Toss Scrapy One: Environment beta0.3 (2013-2-4 Update)

Posted in: http://www.ballooncat.com/scrapy-setup.html

Recently in-depth toss Scrapy, first released the environment to build a tutorial, and then released more about the scrapy configuration, extension aspects of the tutorial. For the beta version, some of the details may differ in the actual environment.

Objective:

1. System centos5.5, python2.4 installed by default, need to upgrade to python2.7+

The installation tutorials include:

Sqlite3 + python2.7.3 + mysql5.1.55 + sphinx2.0.6 + easy_install + python_mysql + scrapy + nginx + uwsgi + Redis Install g Uide

Installation Preparation Work


1. Install the development Kit

Yum install gcc gcc-c++ zlib zlib-devel make cmake libxml2 libxslt-devel openssl-devel imake automake libtool python-devel Ncurses-devel

Start building the environment:

pre_2. Installing Sqlite3
Must be installed before Python is installed

Wget-c Http://www.sqlite.org/sqlite-autoconf-3071501.tar.gztar zxvf sqlite-autoconf-3071501.tar.gz CD Sqlite-autoconf-3071501./configure Makemake INSTALLCD ~

2. Installing python2.7.3

Wget-c Http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgztar zxvf python-2.7.3.tgz cd python-2.7.3./ Configuremake && make Installmv/usr/bin/python/usr/bin/python_old_2.4ln-s/usr/local/bin/python2.7/usr/bin /python python-v (Show 2.7.3 version is installed successfully) Vim/usr/bin/yum modify #!/usr/bin/python to #!/usr/bin/python2.4cd ~

3. Installing Easy_install

Wget-q http://peak.telecommunity.com/dist/ez_setup.py python ez_setup.py

4. Installing mysql5.1.55

Wget-c Http://downloads.mysql.com/archives/mysql-5.1/mysql-5.1.55.tar.gztar ZXVF MYSQL-5.1.55.TAR.GZCD Mysql-5.1.55sh build/autorun.sh./configure--prefix=/usr/local/mysql--with-charset=utf8--with-extra-charset=all- Enable-thread-safe-client-enable-assembler--with-readline--with-big-tables--with-named-curses-libs=/usr/lib/ Libncursesw.so.5make && make Installln-s/usr/local/mysql/bin/mysql/usr/local/bin/mysqlln-s/usr/local/ Mysql/bin/mysqladmin/usr/local/bin/mysqladmingroupadd mysqluseradd-g MySQL mysqlchown-r mysql.mysql/usr/local/ Mysql/cd/usr/local/mysqlmkdir varchown-r root.mysql. Chown-r mysql/usr/local/mysql/var/cp/root/mysql-5.1.55/ Support-files/mysql.server/etc/init.d/mysqldchmod 700/etc/init.d/mysqldchkconfig--add mysqldchkconfig mysqld ONCP/ root/mysql-5.1.55/support-files/my-medium.cnf/etc/my.cnf/usr/local/mysql/bin/mysql_install_db--user=mysql-- Datadir=/usr/local/mysql/var/etc/init.d/mysqld start

mysqladmin-u root password 123123mysql-u root-p-s/tmp/mysql.sockcd ~

5. Installing Sphinx

Wget-c Http://sphinxsearch.com/files/sphinx-2.0.6-release.tar.gztar ZXVF SPHINX-2.0.6-RELEASE.TAR.GZCD Sphinx-2.0.6-release./configuremake && make INSTALLCD/USR/LOCAL/ETC/CP sphinx.conf.dist SPHINX.CONFCD ~

6. Installing Mysql_python

Wget-c http://downloads.sourceforge.net/project/mysql-python/mysql-python-test/1.2.3c1/ Mysql-python-1.2.3c1.tar.gz?use_mirror=nchctar xzvf mysql-python-1.2.3c1.tar.gz CD mysql-python-1.2.3c1ln-s/usr/ Local/mysql/bin/mysql_config/usr/local/bin/mysql_configln-s/usr/local/mysql/lib/mysql/libmysqlclient*/usr/ Libldconfigpython setup.py INSTALLCD ~

7. Installing Scrapy

Wget-c Http://lxml.de/files/lxml-3.0.1.tgztar zxvf lxml-3.0.1.tgz cd lxml-3.0.1python setup.py installcd ~wget-c http:/ /pypi.python.org/packages/source/t/twisted/twisted-12.2.0.tar.bz2#md5=9a321b904d01efd695079f8484b37861tar JXVF TWISTED-12.2.0.TAR.BZ2 CD Twisted-12.2.0

Python setup.py Install

wget http://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.12.tar.gz

Tar zxvf pyopenssl-0.12.tar.gz

CD pyOpenSSL-0.12

Python setup.py installeasy_install SCRAPYCD ~

8. Installing UWSGI

Wget-c Http://projects.unbit.it/downloads/uwsgi-1.4.3.tar.gztar zxvf uwsgi-1.4.3.tar.gz cd Uwsgi-1.4.3python setup.py INSTALLCD ~

9. Installing Redis

wget https://redis.googlecode.com/files/redis-2.6.7.tar.gztar zxvf redis-2.6.7.tar.gzcd redis-2.6.7make && Make Installeasy_install Redis

10. Installing Nginx

wget Ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.30.tar.gztar zxvf pcre-8.30.tar.gz cd pcre-8.30./ Configure make && make install

wget Http://sourceforge.net/projects/libpng/files/zlib/1.2.6/zlib-1.2.6.tar.gz/downloadtar ZXVF zlib-1.2.6.tar.gz CD zlib-1.2.6./configure make && make install

wget Http://nginx.org/download/nginx-1.1.9.tar.gztar zxvf nginx-1.1.9.tar.gz cd nginx-1.1.9mkdir-p/var/tmp/nginx./ Configure--prefix=/usr/local/nginx--pid-path=/var/run/nginx.pid--lock-path=/var/lock/nginx.lock--with-http_ssl _module--with-http_dav_module--with-http_flv_module--with-http_realip_module--with-http_gzip_static_module-- With-http_stub_status_module--with-mail--with-mail_ssl_module--with-pcre=. /pcre-8.30--with-zlib=. /zlib-1.2.6--with-debug--http-client-body-temp-path=/var/tmp/nginx/client--http-proxy-temp-path=/var/tmp/nginx /proxy--http-fastcgi-temp-path=/var/tmp/nginx/fastcgi--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi-- http-scgi-temp-path=/var/tmp/nginx/scgi make && make Installvim/etc/init.d/nginx

Input:

#!/bin/bash  #  #chkconfig:-  #description: Nginx is a world Wide Web server.  #processname: Nginx    nginx=/usr/local/nginx/sbin/nginx  conf=/usr/local/nginx/conf/nginx.conf case $    In    start)   echo-n "Starting Nginx"   $nginx-C $conf   echo "Done"    ;;      Stop)   echo-n "Stopping nginx"   killall-9 nginx   echo "Done"    ;      Test)   $nginx-T-c $conf    ;;    Reload)   echo-n "Reloading nginx"   PS auxww | grep nginx | grep Master | awk ' {print $} ' | Xargs kill-hup   echo "Done"    ;;    Restart) $  Stop $  start    ;      Show)   ps-aux|grep nginx    ;;      *)   Echo-n "Usage: $ start|restart|reload|stop|test|show}"    ;;    Esac

Chkmod +x/etc/init.d/nginxchkconfig--add nginx  chkconfig nginx on

11. Configure Sphinx:
Create scrapy.conf under/usr/local/sphinx/etc, write:

SOURCE Scrapy_source{type        = mysqlsql_host    = Localhostsql_user    = Rootsql_pass    = 123123sql_db      = Pythonsql_port    = 3306sql_query_pre   = SET NAMES utf8sql_query       = SELECT id,title,keywords,descrip,body from ' Data '}index scrapy_index{source      = Scrapy_sourcepath        =/usr/local/sphinx/var/data/scrapy_indexdocinfo     = Externmlock       = 0morphology  = Nonemin_word_len    = 1html_strip  = 0}indexer{mem_limit = 128M} Searchd{listen           = 9312read_timeout     = 5max_children     = 30max_matches      = 1000seamless_rotate  = 0preopen_indexes  = 0unlink_old       = 1pid_file         =/usr/local/sphinx/var/log/searchd.pidlog              =/usr/ Local/sphinx/var/log/searcd.logquery_log        =/usr/local/sphinx/var/log/query.log}

To create an index:

/usr/local/sphinx/bin/indexer-c. /etc/scrapy.conf--all

To re-create the index:

/usr/local/sphinx/bin/indexer-c. /etc/scrapy.conf--all--rotate

Search test:

/usr/local/sphinx/bin/search-c. /etc/scrapy.conf CentOS

beta0.2:

1. Fix a few development kit missing issues

2. Fix Scrapy PYOPENSSL issues during installation

3. Fix MySQL Installation step sequencing issue

beta0.3:

1. Fix a python MySQL extension does not find the MySQL library file when executing the problem

2. The omission of the fix Python redis extension installation

Deep Toss Scrapy One: Environment beta0.3 (2013-2-4 Update)