Deploy pyspider in centos7 in a distributed manner,

Deploy pyspider in centos7 in a distributed manner,

1. Set up the environment:

Linux centos-linux.shared 3.10.0-123. el7.x86 _ 64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Python version: Python 3.5.1

1. Set up the python3 environment:

After trying, I chose the integrated environment Anaconda.

1.1.1. Compile

# Download the dependency yum install-y ncurses-devel openssl-devel zlib-devel gcc make glibc-devel libffi-devel glibc-static glibc-utils sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-deve # download python version wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz# or use domestic source wget http://mirrors.sohu.com/python/3.5.1/Python-3.5.1.tgzmv Python-3.5.1.tgz/usr/local/src; cd/usr/local/src # unzip tar-zxf Python-3.5.1.tgz; cd Python-3.5.1 # compile and install. /configure -- prefix =/usr/local/python3.5 -- enable-sharedmake & make install # create a soft link ln-s/usr/local/python3.5/bin/python3/usr/bin /python3echo "/usr/local/python3.5/lib">/etc/ld. so. conf. d/python3.5.confldconfig # verify python3python3 # Python 3.5.1 (default, Oct 9 2016, 11:44:24) # [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux # Type "help ", "copyright", "credits" or "license" for more information. #>>># pip/usr/local/python3.5/bin/pip3 install -- upgrade pipln-s/usr/local/python3.5/bin/pip/usr/bin/pip # myself installing pip reinstalls wget https://bootstrap.pypa.io/get-pip.py -- no-check-certificatepython get-pip.py

1.1.2. Integrated Environment anaconda

# Integrated environment anaconda (recommended) wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh# can be directly installed./Anaconda3-4.2.0-Linux-x86_64.sh # if an error occurs, it may be decompression failure yum install bzip2

1. 2. Install mariaDB

# Install yum-y install mariadb-server # start systemctl start mariadb # set to start systemctl enable mariadb # Set the password to null by default mysql_secure_installation # log on to mysql-u root-p # create a user sets the account password to CREATE user 'user _ name' @ 'localhost' identified by 'user _ pass '; grant all privileges on *. * TO 'user _ name' @ 'localhost' with grant option; CREATE user 'user _ name' @ '% 'identified BY 'user _ pass'; grant all privileges on *. * TO 'user _ name' @ '%' with grant option;

1. 3. Install pyspider

I use Anaconda

# Build a virtual environment sbird python version 3. * conda create-n sbird python = 3 * # enter the environment source activate sbird # install pyspiderpip install pyspider # Error # it does not exist. the exported locale is "en_US.UTF-8" but it is not supported # The execution can be written. bashrcexport LC_ALL = en_US.utf-8export LANG = en_US.utf-8 # ImportError: pycurl: libcurl link-time version (7.29.0) is older than compile-time version (7.49.0) conda install pycurl # exit source deactivate sbird # disable firewall systemctl stop firewalld if localhost cannot be accessed in the VM. service ######### run the source code directly =============== mkdir git; cd git # download git clone https://github.com/binux/pyspider.git# installation/root/anaconda3/envs/sbird/bin/python/root/git/pyspider/run. py

Other methods

# Build a virtual environment pip install virtualenvmkdir python; cd python # create a virtual environment pyenv3virtualenv-p/usr/bin/python3 pyenv3 # enter the virtual environment to activate the environment cd pyenv3/source. /bin/activatepip install pyspider # If pycurl reports an error, yum install libcurl-devel # continue pip install pyspider # disable deactivate

I recommend using anaconda for installation.

If an error occurs during pyspider running, see the installation section of anaconda. Now, visit localhost: 5000 to see the page.

. Install Supervisor

# Install yum install supervisor-y # Add Alibaba's epel source vim/etc/yum if retrieval fails. repos. d/epel. repo # Add the following content [epel] name = Extra Packages for Enterprise Linux 7-$ basearchbaseurl = paipackages for Enterprise Linux 7-$ basearch-Debugbaseurl = paipackages for Enterprise Linux 7-$ basearch-Sourcebaseurl = http://mirrors.aliyun.com/epel/7/SRPMShttp://mirrors.aliyuncs.com/epel/7/SRPMSfailovermethod=priorityenabled=0gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7gpgcheck=0# install yum install supervisor-y # test whether echo_supervisord_conf is installed successfully

1.4.1.Supervisor usage

Supervisord # Start supervisorctl on the server side of the supervisor # Start the command line window of the supervisor # assume that the creation process pyspider01vim/etc/supervisord is created. d/pyspider01.ini # Write the following content to [program: pyspider01] command =/root/anaconda3/envs/sbird/bin/python/root/git/pyspider/run. pydirectory =/root/git/pyspideruser = rootprocess_name = % (program_name) sautostart = trueautorestart = truestartsecs = train =/pyspider/supervisor/train # Heavy Load supervisorctl reload # start supervisorctl start pyspider01 # supervisord-c/etc/ supervisord. conf # view status supervisorctl status # output pyspider01 RUNNING pid 4026, uptime 0:02:40 # disable supervisorctl shutdown

1. 5. Install redis

# Message Queue using redismkdir download; cd downloadwget http://download.redis.io/releases/redis-3.2.4.tar.gztar xzf redis-3.2.4.tar.gzcd redis-3.2.4make # or directly yum install yum-y install redis # start systemctl start redis. service # restart systemctl restart redis. service # stop systemctl stop redis. service # view the status of systemctl status redis. service # change the file/etc/redis. confvim/etc/redis. conf # Change daemonize no to daemonize yesbind 127.0.0.1 to bind 10.211.55.22 (current server ip address) # restart redissystemctl restart redis. service

. About Auto-start

# Add the Supervisor to the auto-starting service systemctl enable supervisord. service # Add redis to the auto-starting service systemctl enable redis. service # disable the firewall and enable systemctl disable firewalld. service

So far, the runtime environment of a single server of pyspider has been set up and deployed, and localhost: 5000 is started to enter the web interface.

You can also write a script to view the running status in/pyspider/supervisor/pyspider01.log.

2. distributed deployment

Name the server you Just configured as centos01 and deploy two servers, centos02 and centos03, respectively.

As follows:

Server Name ip address description

centos01 10.211.55.22 redis,mariaDB, schedulercentos02 10.211.55.23 fetcher, processor, result_worker,phantomjscentos03 10.211.55.24 fetcher, processor,,result_worker,webui

2.1.centos01

Go to centos01, and set up the basic environment. First, edit the configuration file/pyspider/config. json.

{ "taskdb": "mysql+taskdb://user_name:user_pass@10.211.55.22:3306/taskdb", "projectdb": "mysql+projectdb://user_name:user_pass@10.211.55.22:3306/projectdb", "resultdb": "mysql+resultdb://user_name:user_pass@10.211.55.22:3306/resultdb", "message_queue": "redis://10.211.55.22:6379/db", "logging-config": "/pyspider/logging.conf", "phantomjs-proxy":"10.211.55.23:25555", "webui": {  "username": "",  "password": "",  "need-auth": false,  "host":"10.211.55.24",  "port":"5000",  "scheduler-rpc":"http:// 10.211.55.22:5002",  "fetcher-rpc":"http://10.211.55.23:5001" }, "fetcher": {  "xmlrpc":true,  "xmlrpc-host": "0.0.0.0",  "xmlrpc-port": "5001" }, "scheduler": {  "xmlrpc":true,  "xmlrpc-host": "0.0.0.0",  "xmlrpc-port": "5002" }}

Run the following command:

/Root/anaconda3/envs/sbird/bin/python/root/git/pyspider/run. py-c/pyspider/config. json scheduler # error ImportError: No module named 'mysql' # download mysql-connector-pythoncd ~ /Git clone https://github.com/mysql/mysql-connector-python.git# install source activate sbirdcd mysql-connector-pythonpython setup. py install # install redispip install redissource deactivate # run/root/anaconda3/envs/sbird/bin/python/root/git/pyspider/run. py-c/pyspider/config. json schedstart# output OK [I 161010 15:57:25 scheduler: 644] schedstartstarting... [I 161010 15:57:25 scheduler: 779] scheduler. xmlrpc listening on 0.0.0.0: 5002 [I 161010 15:57:25 scheduler: 583] in 5 m: new: 0, success: 0, retry: 0, failed: 0

After running successfully, you can directly change/etc/supervisord. d/pyspider01.ini as follows:

[Program: pyspider01] command =/root/anaconda3/envs/sbird/bin/python/root/git/pyspider/run. py-c/pyspider/config. json schedulerdirectory =/root/git/pyspideruser = rootprocess_name = % (program_name) sautostart = trueautorestart = truestartsecs = 3redirect_stderr = direction = 10stdout_logfile =/pyspider/supervisor/pyspider01.log # reload supervisorctl reload # view status supervisorctl status

Deployment of centos01 is complete.

2.2.centos02

In centos02, run result_worker, processor, phantomjs, and fetcher.

Create files respectively:

/etc/supervisord.d/result_worker.ini[program:result_worker]command   = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json result_workerdirectory  = /root/git/pyspideruser     = rootprocess_name = %(program_name)sautostart  = trueautorestart = truestartsecs  = 3redirect_stderr     = truestdout_logfile_maxbytes = 500MBstdout_logfile_backups = 10stdout_logfile     = /pyspider/supervisor/result_worker.log/etc/supervisord.d/processor.ini[program:processor]command   = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json processordirectory  = /root/git/pyspideruser     = rootprocess_name = %(program_name)sautostart  = trueautorestart = truestartsecs  = 3redirect_stderr     = truestdout_logfile_maxbytes = 500MBstdout_logfile_backups = 10stdout_logfile     = /pyspider/supervisor/processor.log/etc/supervisord.d/phantomjs.ini[program:phantomjs]command   = /pyspider/phantomjs --config=/pyspider/pjsconfig.json /pyspider/phantomjs_fetcher.js 25555directory  = /root/git/pyspideruser     = rootprocess_name = %(program_name)sautostart  = trueautorestart = truestartsecs  = 3redirect_stderr     = truestdout_logfile_maxbytes = 500MBstdout_logfile_backups = 10stdout_logfile     = /pyspider/supervisor/phantomjs.log/etc/supervisord.d/fetcher.ini[program:fetcher]command   = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json fetcherdirectory  = /root/git/pyspideruser     = rootprocess_name = %(program_name)sautostart  = trueautorestart = truestartsecs  = 3redirect_stderr     = truestdout_logfile_maxbytes = 500MBstdout_logfile_backups = 10stdout_logfile     = /pyspider/supervisor/fetcher.log

Create pjsconfig. json in the pyspider directory

{ /*--ignore-ssl-errors=true */ "ignoreSslErrors": true, /*--ssl-protocol=true */ "sslprotocol": "any", /* Same as: --output-encoding=utf8 */ "outputEncoding": "utf8", /* persistent Cookies. */ /*cookiesfile="e:/phontjscookies.txt",*/ cookiesfile="pyspider/phontjscookies.txt", /* load image */ autoLoadImages = false}

Download phantomjs to the/pyspider/folder and copy git/pyspider/fetcher/phantomjs_fetcher.js to phantomjs_fetcher.js.

# Reload supervisorctl reload # view the supervisorctl status # outputfetcher RUNNING pid 3446, uptime 0: 00: 07 phantomjs RUNNING pid 3448, uptime 0: 00: 07 processor RUNNING pid 3447, uptime 0: 00: 07result_worker RUNNING pid 3445, uptime 0:00:07

Deployment of centos02 is complete.

2.3.centos03

Deploy these three processes, like fetcher, processor, result_worker, and centos02. This server mainly adds webui

Create a file:

/Etc/supervisord. d/webui. ini [program: webui] command =/root/anaconda3/envs/sbird/bin/python/root/git/pyspider/run. py-c/pyspider/config. json webuidirectory =/root/git/pyspideruser = rootprocess_name = % (program_name) sautostart = trueautorestart = truestartsecs = login =/pyspider/supervisor/webui. log # reload supervisorctl reload # view the supervisorctl status # outputfetcher RUNNING pid 2724, uptime 0: 00: 07 processor RUNNING pid 2725, uptime 0: 00: 07result_worker RUNNING pid 2723, uptime 0: 00: 07 webui RUNNING pid 2726, uptime 0:00:07

3. Summary

Access http: // 10.211.55.24: 5000.