Powerful web crawler system: Pyspider

Pyspider: A powerful web crawler system written by a nation with powerful WebUI. Written in Python language, distributed architecture, supports multiple database backend, powerful WebUI support Script Editor, Task Monitor, project manager and result viewer.

1. Build the Environment:

System version: Linux centos-linux.shared 3.10.0-123.el7.x86_64 #1 SMP Mon June 12:09:22 UTC x86_64 x86_64 x86_64 gnu/linux

Python version: Python 3.5.1

1.1. Build Python3 Environment:

I am trying to choose an integrated environment after the Anaconda

1.1.1. Compiling

# download Dependent yum install-y ncurses-devel OpenSSL openssl-devel zlib-devel gcc make glibc-devel libffi-devel glibc-static glibc-u TILs sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-deve# download python version wget https:// www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz# or use domestic source wget HTTP://MIRRORS.SOHU.COM/PYTHON/3.5.1/ PYTHON-3.5.1.TGZMV python-3.5.1.tgz/usr/local/src;cd/usr/local/src# decompression TAR-ZXF python-3.5.1.tgz;cd Python-3.5.1# Compile the installation./configure--prefix=/usr/local/python3.5--enable-sharedmake && make install# creating a soft link ln-s/usr/local/ Python3.5/bin/python3/usr/bin/python3echo "/usr/local/python3.5/lib" >/etc/ld.so.conf.d/ python3.5.confldconfig# validation python3python3# Python 3.5.1 (default, Oct 9, 11:44:24) # [GCC 4.8.5 20150623 (Red Hat 4.8.5 -4)] on linux# Type ' help ', ' copyright ', ' credits ' or ' license ' for more information.# >>># Pip/usr/local/python3 .5/BIN/PIP3 Install--upgrade pipln-s/usr/local/python3.5/bin/pip/usr/bin/pip# I'm having problems installing PIP reload wget https://bootstrap.pypa.io/get-pip.py--no-check-certificatepython get-pip.py 

1.1.2. Integrated Environment Anaconda

# Integrated Environment Anaconda (recommended) wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh# can be installed directly. anaconda3-4.2.0-linux-x86_64.sh# if error, may be decompression failed yum install bzip2

1.2. Installing MARIADB

# installation Yum-y Install mariadb mariadb-server# boot systemctl start mariadb# set to boot systemctl enable mariadb# Configure password default to null MYSQL_SECU re_installation# sign-in mysql-u root-p# create a user to set the account password created by username ' user_name ' @ ' localhost ' identified by ' user_pass '; GRANT all privileges on *. * to ' user_name ' @ ' localhost ' with GRANT OPTION; CREATE USER ' user_name ' @ '% ' identified by ' user_pass '; GRANT all privileges on *. * to ' user_name ' @ '% ' with GRANT OPTION;

1.3. Installing Pyspider

I use Anaconda

# Build a virtual Environment Sbird python version 3.*conda create-n sbird python=3*# Enter the environment source activate sbird# installation Pyspiderpip install pyspider# error # It does not exist. The exported locale is "en_US. UTF-8 "But it's not supported# performs writable. Bashrcexport Lc_all=en_us.utf-8export Lang=en_us.utf-8#importerror:pycurl:libcur L Link-time Version (7.29.0) is older than Compile-time version (7.49.0) Conda install pycurl# exit source Deactivate sbird# if Unable to access localhost:5000 in virtual machine can shut down firewall systemctl stop firewalld.service######## #直接运行源码 ==============mkdir git;cd git# download git Clone https://github.com/binux/pyspider.git# Installation/root/anaconda3/envs/sbird/bin/python/root/git/pyspider/run.py

Other methods

# Build a virtual environment pip install Virtualenvmkdir PYTHON;CD python# Create a virtual environment pyenv3virtualenv-p/usr/bin/python3 pyenv3# Enter the virtual Environment Activation Environment CD Pye Nv3/source./bin/activatepip Install pyspider# if pycurl error yum install libcurl-devel# continue pip install pyspider# close Deactivate

I recommend the installation by means of Anaconda

If an error occurs during Pyspider operation, refer to the Anaconda Installation section and visit localhost:5000 to see the page.

1.4. Installing Supervisor

# Installing Yum Install supervisor-y# if unable to retrieve add Ali's Epel source vim/etc/yum.repos.d/epel.repo# add the following [Epel]name=extra Packages for Enter Prise Linux 7-$basearchbaseurl =http://mirrors.aliyun.com/epel/7/$basearchhttp://mirrors.aliyuncs.com/epel/7/$ basearchfailovermethod=priorityenabled=1gpgcheck=0gpgkey=file:///etc/pki/rpm-gpg/rpm-gpg-key-epel-7[ Epel-debuginfo]name=extra Packages for Enterprise Linux 7-$basearch-debugbaseurl=http://mirrors.aliyun.com/epel/7/$ basearch/debughttp://mirrors.aliyuncs.com/epel/7/$basearch/debugfailovermethod=priorityenabled=0gpgkey=file:// /etc/pki/rpm-gpg/rpm-gpg-key-epel-7gpgcheck=0[epel-source]name=extra Packages for Enterprise Linux 7-$basearch- Sourcebaseurl=http://mirrors.aliyun.com/epel/7/srpmshttp://mirrors.aliyuncs.com/epel/7/srpmsfailovermethod= priorityenabled=0gpgkey=file:///etc/pki/rpm-gpg/rpm-gpg-key-epel-7gpgcheck=0# Installing yum Install supervisor-y# Test whether the installation was successful echo_supervisord_conf

1.4.1.Supervisor usage

Supervisord   #supervisor的服务器端部分 start supervisorctl  #启动supervisor的命令行窗口 # Suppose you create a process pyspider01vim/etc/ Supervisord.d/pyspider01.ini# writes the following [Program:pyspider01]command   =/root/anaconda3/envs/sbird/bin/python/root /git/pyspider/run.pydirectory  =/root/git/pyspideruser     = Rootprocess_name =% (program_name) Sautostart  = Trueautorestart = Truestartsecs  = 3redirect_stderr     = Truestdout_logfile_maxbytes = 500mbstdout_logfile_ backups = 10stdout_logfile     =/pyspider/supervisor/pyspider01.log# overload supervisorctl reload# start Supervisorctl startup Pyspider01# can also start supervisord-c/etc/supervisord.conf# View status supervisorctl status# output Pyspider01            RUNNING  PID 4026, uptime 0:02:40# close supervisorctl shutdown

1.5. Installing Redis

# Message Queuing with Redismkdir download;cd downloadwget Http://download.redis.io/releases/redis-3.2.4.tar.gztar xzf REDIS-3.2.4.TAR.GZCD redis-3.2.4make# or Direct yum installation yum-y install redis# start systemctl start redis.service# restart Systemctl Restart redis.service# Stop Systemctl stop redis.service# View status systemctl status redis.service# change file/etc/redis.confvim/etc/ redis.conf# Change the content daemonize No to daemonize yesbind 127.0.0.1 to bind 10.211.55.22 (current server IP) # restart Redissystemctl restart Redis. Service

1.6. About self-priming

# Supervisor added to self-starting service Systemctl enable supervisord.service# Redis added to self-boot service Systemctl enable redis.service# Turn off firewall self-booting systemctl disable Firewalld.service

At this point, Pyspider a single server running environment to build and deploy, start localhost:5000 into the Web interface.

You can also write scripts to run and view the running state in/pyspider/supervisor/pyspider01.log.

2. Distributed Deployment

Just configure the server, name it centos01, according to this configuration, and then deploy two centos02, Centos03 respectively.

As follows:

Server Name IP description

Centos01 10.211.55.22 redis,mariadb, Schedulercentos02 10.211.55.23 fetcher, processor, Result_worker, Phantomjscentos03 10.211.55.24 Fetcher, Processor,,result_worker,webui

2.1.centos01

Enter the server centos01, after the first step, the basic environment has been set up, first edit the configuration file/pyspider/config.json

{"TASKDB": "Mysql+taskdb://user_name:user_pass@10.211.55.22:3306/taskdb", "Projectdb": "Mysql+projectdb://user_ Name:user_pass@10.211.55.22:3306/projectdb "," Resultdb ":" mysql+resultdb://user_name:user_pass@10.211.55.22:3306 /resultdb "," Message_queue ":" redis://10.211.55.22:6379/db "," Logging-config ":"/pyspider/logging.conf "," Phantomjs-proxy ":" 10.211.55.23:25555 "," WebUI ": {  " username ":" ",  " password ":" ",  " Need-auth ": false,  "host": "10.211.55.24",  "port": "The", "  scheduler-rpc": "/http/10.211.55.22:5002",  " Fetcher-rpc ":" http://10.211.55.23:5001 "}," Fetcher ": {  " xmlrpc ": True,  " Xmlrpc-host ":" 0.0.0.0 ",  " Xmlrpc-port ":" 5001 "}," Scheduler ": {  " xmlrpc ": True,  " Xmlrpc-host ":" 0.0.0.0 ",  " Xmlrpc-port ":" 5002 " }}

Try running under:

/root/anaconda3/envs/sbird/bin/python/root/git/pyspider/run.py-c/pyspider/config.json scheduler# Error Importerror: No module named ' mysql ' # download MYSQL-CONNECTOR-PYTHONCD ~/git/git clone https://github.com/mysql/ mysql-connector-python.git# Installing source Activate SBIRDCD Mysql-connector-pythonpython setup.py install# installation Redispip Install Redissource deactivate# Run/root/anaconda3/envs/sbird/bin/python/root/git/pyspider/run.py-c/pyspider/ Config.json scheduler# output ok[i 161010 15:57:25 scheduler:644] Scheduler starting ... [I 161010 15:57:25 scheduler:779] scheduler.xmlrpc listening on 0.0.0.0:5002[i 161010 15:57:25 scheduler:583] in 5m:new:0 , success:0,retry:0,failed:0

After successful operation, the/etc/supervisord.d/pyspider01.ini can be changed directly as follows:

[Program:pyspider01]command   =/root/anaconda3/envs/sbird/bin/python/root/git/pyspider/run.py-c/pyspider/ Config.json schedulerdirectory  =/root/git/pyspideruser     = Rootprocess_name =% (program_name) Sautostart  = Trueautorestart = Truestartsecs  = 3redirect_stderr     = Truestdout_logfile_maxbytes = 500mbstdout_logfile_ backups = 10stdout_logfile     =/pyspider/supervisor/pyspider01.log# overload supervisorctl reload# View status Supervisorctl Status

CENTOS01 deployment is complete.

2.2.centos02

In CENTOS02, you need to run Result_worker, processor, PHANTOMJS, Fetcher

Create files separately:

/etc/supervisord.d/result_worker.ini[program:result_worker]command =/root/anaconda3/envs/sbird/bin/python/root/ Git/pyspider/run.py-c/pyspider/config.json result_workerdirectory =/root/git/pyspideruser = Rootprocess_name =% (p Rogram_name) Sautostart = Trueautorestart = Truestartsecs = 3redirect_stderr = Truestdout_logfile_maxbytes = 500MBstd out_logfile_backups = 10stdout_logfile =/pyspider/supervisor/result_worker.log/etc/supervisord.d/processor.ini[ Program:processor]command =/root/anaconda3/envs/sbird/bin/python/root/git/pyspider/run.py-c/pyspider/ Config.json processordirectory =/root/git/pyspideruser = Rootprocess_name =% (program_name) Sautostart = Trueautores Tart = Truestartsecs = 3redirect_stderr = Truestdout_logfile_maxbytes = 500mbstdout_logfile_backups = 10stdout_logfil E =/pyspider/supervisor/processor.log/etc/supervisord.d/phantomjs.ini[program:phantomjs]command =/pyspider/ Phantomjs--config=/pyspider/pjsconfig.json/pyspider/phantOmjs_fetcher.js 25555directory =/root/git/pyspideruser = Rootprocess_name =% (program_name) Sautostart = Trueautores Tart = Truestartsecs = 3redirect_stderr = Truestdout_logfile_maxbytes = 500mbstdout_logfile_backups = 10stdout_logfil E =/pyspider/supervisor/phantomjs.log/etc/supervisord.d/fetcher.ini[program:fetcher]command =/root/anaconda3/ Envs/sbird/bin/python/root/git/pyspider/run.py-c/pyspider/config.json fetcherdirectory =/root/git/pyspideruser = Rootprocess_name =% (program_name) Sautostart = Trueautorestart = Truestartsecs = 3redirect_stderr = Truestdout_logfi Le_maxbytes = 500mbstdout_logfile_backups = 10stdout_logfile =/pyspider/supervisor/fetcher.log

Establish Pjsconfig.json in the Pyspider directory

{/*--ignore-ssl-errors=true */"ignoresslerrors": true,/*--ssl-protocol=true */"Sslprotocol": "Any",/* Same as:--OUTP Ut-encoding=utf8 */"outputencoding": "UTF8",/* persistent Cookies. *//*cookiesfile= "e:/phontjscookies.txt", */cookiesfile= "pyspider/phontjscookies.txt",/* Load image */ Autoloadimages = False}

Download Phantomjs to/pyspider/folder, copy Git/pyspider/pyspider/fetcher/phantomjs_fetcher.js to Phantomjs_fetcher.js

# reload Supervisorctl reload# View status supervisorctl status# outputfetcher             RUNNING  pid 3446, uptime 0:00:07phantomjs            RUNNING  pid 3448, uptime 0:00:07processor            RUNNING  pid 3447, uptime 0:00:07result_worker          RUNNING  pid 3445, uptime 0:00:07

CENTOS02 deployment is complete.

2.3.centos03

Deploy these three processes Fetcher, processor, Result_worker and CENTOS02, this server is mainly based on the previous add WebUI

To create a file:

/etc/supervisord.d/webui.ini[program:webui]command   =/root/anaconda3/envs/sbird/bin/python/root/git/ Pyspider/run.py-c/pyspider/config.json webuidirectory  =/root/git/pyspideruser     = Rootprocess_name =% ( program_name) Sautostart  = Trueautorestart = Truestartsecs  = 3redirect_stderr     = Truestdout_logfile_ MaxBytes = 500mbstdout_logfile_backups = 10stdout_logfile     =/pyspider/supervisor/webui.log# overload Supervisorctl reload# View status Supervisorctl status# outputfetcher             RUNNING  pid 2724, uptime 0:00:07processor            RUNNING  PID 2725, uptime 0:00:07result_worker          RUNNING  pid 2723, uptime 0:00:07webui              RUNNING  pid 2726, Uptime 0:00:07

3. Summary

"Recommended"

1. Python Free video tutorial

2. Python Learning Manual

3. Python Object-oriented video tutorial