The reason
Recently I used crawlers to crawl some websites and found that sometimes the network is not stable. After switching the network, the results can be crawled.
So there is the idea of building an agent. Usually, the agents provided on the Internet have a short effective time and are very unstable.
In order to have a few stable agents to use, you must either pay or build it yourself. So finally chose to build an agent on
the server.
Alibaba Cloud Simple Application Server: Anti COVID-19 SME Enablement Program
$300 coupon package for all new SMEs and a $500 coupon for paying customers.
How to create
After looking up some information, I found that Squid is usually used to
build a proxy server on Linux, but after a lot of toss, I found that it is relatively cumbersome to build a proxy server with Squid. So finally chose Shadowsocks + privoxy to realize the simple proxy mode of sock5 proxy to http proxy.
Implementation steps
1. First install the python-pip module, directly execute easy_install pip or directly apt install #The purpose of this step is to install shadowsocks next
2. Install python-shadowsocks module pip install shadowscoks
3. Background process
sudo ssserver -p 8338 -k password -m rc4-md5 --user nobody -d start
The steps in this step are to let shadowsock listen to port 8338 to implement socks5 proxy
Everything is normal, access directly through the socks5 proxy
curl -socks5 http://you.domain:8338 http://www.baidu.com
ps: Parameter setting, please refer to https://pypi.org/project/shadowsocks/ manual
4. Shadowsock can only complete socks5 proxy, if we need to implement http proxy, then we can only convert http protocol and socks5 protocol. Privoxy can achieve this function.
- Install privoxy apt-get install privoxy
- Edit the configuration file located at /etc/privoxy/config and modify the following parameters
listen-address 0.0.0.0:8118 #Indicates that the external network can also be used as an http proxy through the local IP
forward-socks5 / 127.0.0.1:8338.
After the startup is complete, you can access it by curl -x http://you.host http://www.baidu.com
Through the above steps, the basic functions of
the proxy server can be realized, the configuration is simple, and the personal use is basically satisfied.
If you have higher requirements for a proxy server, you can use squid to build a proxy server.
