Enable scrapy to support socks proxy and proxy pool in disguise

Source: Internet
Author: User

In the previous article, I wrote how to enable scrapy to Support HTTP proxy.

But scrapy does not support socks proxy by default. Sometimes the pure HTTP proxy is easily intercepted by g f w, and proxy is required to collect websites blocked from abroad. Okay, capability is forced by demand.


The following describes a solution.

Deploy a Linux VPs in the United States or Hong Kong


Debian as an Example

Install necessary components

Apt-Get install build-essential Autoconf libtool libssl-dev gcc-y


Install git

Apt-Get install Git-y


Download and compile the shadowsocks-libev source code package

Git clone https://github.com/madeye/shadowsocks-libev.git

 

CD shadowsocks-libev

 

./Configure

 

Make & make install


Run shadowsocks

/Usr/local/bin/SS-server-s 0.0.0.0-P port-K password-M encryption mode &


Local server (Windows or Linux)

Go http://dl.chenyufei.info/shadowsocks/

Download the shadowsocks client of the corresponding system


In addition, the shadowsocks client can specify multiple servers, which is equivalent to a proxy pool.


The most critical step is to convert the socks proxy into an HTTP proxy.


3 proxy is recommended here, And the homepage is www.3proxy.ru.

3proxy supports windows, Linux, and other platforms. Download and install the SDK directly in windows.


3proxy's typical configuration file

Nscache 65536

Timeouts 1 5 30 60 180 1800 15 60

Daemon

# Service

# External IP

Internal 127.0.0.1

Auth iponly

Allow 127.0.0.1

Parent 1000 SOCKS5 + 127.0.0.1 9999

Proxy-n-a-p1984


127.0.0.1 is the upper-level SOCKS5 proxy server, and 9999 is the proxy server port

1984 indicates the listening port

For more detailed settings, refer to this article.

Http://blog.sunshine-wang.info /? P = 20

Or read the official documentation.


Finally, set the HTTP proxy of scrapy to http: // 127.0.0.1: 1984.

You can.


References

Http://igfw.net/archives/947

Http://www.ouvps.com /? P = 464


Enable scrapy to support socks proxy and proxy pool in disguise

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.