CENTOS7 production Environment IP proxy pool (python)

Recently in the research crawler, need to deploy IP agent pool in front, so in open source China to find proxy pool. can automatically crawl the domestic several free IP proxy website IP, and verify the availability of IP in real time, the database is Ssdb.
IP Proxy Pool Web site:
http://www.data5u.com/
http://www.data5u.com/free/
Http://www.data5u.com/free/gngn/index.shtml
Http://www.data5u.com/free/gnpt/index.shtml
http://www.66ip.cn/
http://www.ip181.com/
Http://www.xicidaili.com/nn
Http://www.xicidaili.com/nt
Http://www.goubanjia.com/free/gngn/index.shtml
http://www.xdaili.cn/ipagent/freeip/getFreeIps?page=1&rows=10

2 yum-y Install git
3 Yum–y Install wget
4 Yum Install curl-devel expat-devel gettext-devel openssl-devel zlib-devel
5 Yum Install gcc perl-extutils-makemaker epel-release gcc-c++
8 cd/usr/src/
wget https://www.kernel.org/pub/software/scm/git/git-2.9.5.tar.gz #安装克隆工具git
Tar-xzf git-2.9.5.tar.gz
CD git-2.9.5
Prefix=/usr/local/git All
Make Prefix=/usr/local/git Install
echo "Export path= $PATH:/usr/local/git/bin" >>/ETC/BASHRC
Source/etc/bashrc
CD.
git clone https://github.com/jhao104/proxy_pool.git #克隆proxy_pool
CD proxy_pool/
Python–v #查看python版本2.7.5
Yum-y Install PYTHON34 #安装python 3.4
wget--no-check-certificate https://bootstrap.pypa.io/get-pip.py
Python3 get-pip.py #安装pip
Pip Install-r requirements.txt #proxy_pool的安装依赖包
cd/usr/local/
git clone https://github.com/ideawu/ssdb.git #克隆SSDB
CD Ssdb
Yum-y Install autoconf
Notoginseng CD deps/snappy-1.1.0/#编译Snappy
/configure.
Make
Cd/usr/local/ssdb
Make #安装SSDB
Make install
Ln-sf/usr/local/ssdb/ssdb-server/usr/local/bin/ssdb-server
Ln-sf/usr/local/ssdb/tools/ssdb-cli/usr/local/bin/ssdb-cli
Ln-sf/usr/local/ssdb/tools/ssdb-dump/usr/local/bin/ssdb-dump
Ln-sf/usr/local/ssdb/tools/ssdb-repair/usr/local/bin/ssdb-repair
Ln-sf/usr/local/ssdb/tools/ssdb.sh/etc/rc.d/init.d/ssdb
Chkconfig--add Ssdb
Chkconfig Ssdb on
Systemctl Stop Firewalld.service #关闭防火墙
Wuyi systemctl Disable Firewalld.service
Firewall-cmd--state
Pip Install--upgrade Pyssdb
Start Proxy_pool Order
1 Cd/usr/local/ssdb
2./ssdb-server-d./ssdb.conf-s start
Note: The SSDB service will generate a ssdb.pid file in the Var directory when it starts, and the file will not be deleted when the Ssdb is accidentally closed, so when the SSDB service is restarted, it will be reported incorrectly, so running the following 2 Lines command can be restarted manually.
./ssdb-server Ssdb.conf-s Stop
./ssdb-server-d./ssdb.conf-s restart

3 Cd/usr/src/proxy_pool/run
4 python3.4 main.py #用python3.4 start

Client Access
http://ip:5010/
Http://ip:5010/get (randomly pick an IP and port)
Http://ip:5010/get_all (pick all available IP and port)

For specific use please refer to Https://github.com/jhao104/proxy_pool
This article provides deployment details under CENOTS7, thanks to contributor and j_hao104 for their selfless dedication!

CENTOS7 production Environment IP proxy pool (python)