Python bot proxy IP Pool implementation method

Source: Internet
Author: User
In the company to do distributed Deep web crawler, set up a stable agent pool service, for thousands of reptiles to provide effective agents, to ensure that each crawler is the corresponding site effective proxy IP, so as to ensure that the crawler fast and stable operation, so you want to use some free resources to engage in a simple proxy pool service.

In the company to do distributed Deep web crawler, set up a stable agent pool service, for thousands of crawlers to provide effective agents, to ensure that each crawler is the corresponding site effective proxy IP, so as to ensure that the crawler fast and stable operation, of course, the company do things can not open source out. However, leisure time preface, so I want to use some free resources to engage in a simple proxy pool service.

1. Questions

Where does proxy IP come from?
Just self-study crawler when there is no proxy IP to the West Thorn, fast Proxy and other free agent of the site to climb, or there are individual agents can use.   Of course, if you have a better proxy interface can also be self-access. Free Proxy collection is also very simple, nothing is: Access page page, regular/xpath extract, save

How to ensure the quality of agents?
Can be sure that the free proxy IP is mostly not available, or else why others provide pay (but the fact is that many agents pay IP is not stable, there are many are not available). So the collection of proxy IP is not directly used, you can write the detection program to use these agents to access a stable website, see if it can be used normally. This process can be either multithreaded or asynchronous, because the detection agent is a slow process.

How are the collected agents stored?
There is a need to recommend a NoSQL database Ssdb with high performance support for multiple data structures for the proxy redis. Supports queue, hash, set, k-v pairs, and supports T-level data. is to do distributed crawler very good intermediate storage tool.

How to make it easier for crawlers to use these proxies?
The answer must be to make a service, Python has so many web frameworks, take one to write an API for the crawler to invoke. This has many advantages, such as: When the crawler found that the agent can not use the active API to delete the proxy IP, when the crawler found that the agent pool IP is not enough to take the initiative to refresh the agent pool. This is more reliable than the test procedure.

2, Agent Pool design

The agent pool consists of four parts:

Proxygetter:
The agent obtains the interface, currently has 5 free proxy source, each call will crawl this 5 website the newest agent to put in the DB, can add the additional proxy to obtain the interface by itself;

Db:
For hosting proxy IPs, now only SSDB is supported. As for why choose Ssdb, we can refer to this article, personally think Ssdb is a good redis alternative, if you have not used Ssdb, installation is also very simple, you can refer to here;

Schedule:
Schedule task users to periodically detect agent availability in DB and remove unavailable proxies. At the same time will actively through the Proxygetter to obtain the latest agent into the DB;

PROXYAPI:
Agent Pool external interface, because now so the agent pool function is relatively simple, spend two hours to see the next flask, happy decision with flask fix. function is to provide the crawler with Get/delete/refresh and other interfaces, easy to use the crawler directly.

[Html_removed] Design

3. Code module

Python's high-level data structures, dynamic types, and dynamic bindings make it ideal for rapid application development and for connecting existing software components as glue language. Using Python to make this proxy IP pool is also very simple, the code is divided into 6 modules:

Api:api interface-related code, the current API is implemented by flask, the code is very simple. Client requests are passed to Flask,flask to invoke the implementation in Proxymanager, including get/delete/refresh/get_all ;

DB: Database related code, the current database is using SSDB. The code is implemented in Factory mode to facilitate future expansion of other types of databases;

Manager: get/delete/refresh/get_all the specific implementation of the interface class, the current agent pool is only responsible for the management proxy, in the future there may be more features, such as Agent and crawler bindings, agent and account binding, etc.;

Proxygetter: Agent to obtain the relevant code, currently grabbed fast agent, Agent 66, there are agents, the West Thorn agent, Guobanjia this five free agent of the site, tested this 5 Web site updated daily available agents only sixty or seventy, of course, also support their expansion agent interface;

Schedule: Timed task related code, now just to implement timed to refresh the code, and verify the available agents, the use of multi-process mode;

Util: Store Some common module methods or functions, including GetConfig : Read the configuration file Config.ini class, ConfigParse : Integrate the overridden ConfigParser class to make it case-sensitive, Singleton : Implement Singleton, LazyProperty : Implement class properties lazy calculation. And so on

Other files: Config file: config.ini, database configuration and proxy get interface configuration, can add new proxy get method in Getfreeproxy, and register in Config.ini to use;

4. Installation

Download code:

git clone git@github.com:jhao104/proxy_pool.git or download zip file directly to Https://github.com/jhao104/proxy_pool

Installation dependencies:

Pip Install-r requirements.txt

Start:

You need to start the timer task separately and API to Config.ini Configure your ssdb to schedule directory: >>>python proxyrefreshschedule.py to the API directory:>>> Python proxyapi.py

5. Use

After the timer task starts, all agents are fetched into the database and verified by the proxy fetch method. The default is repeated every 20 minutes thereafter. After a scheduled task starts for about two minutes, you can see the available agents refreshed in SSDB:

After you start proxyapi.py, you can use the interface in your browser to get the proxy, which is in the browser:
Index page:


Get page:


Get_all page:


Crawler, if you want to use it in the crawler code, you can encapsulate this API as a function to use directly, for example:

Import Requestsdef get_proxy ():  return Requests.get ("http://127.0.0.1:5000/get/"). Contentdef Delete_proxy ( Proxy):  requests.get ("http://127.0.0.1:5000/delete/?proxy={}". Format (proxy)) # your Spider Codedef spider ():  # ....  Requests.get (' https://www.example.com ', proxies={"http": "http://{}". Format (Get_proxy)})  # ....

6. Finally

The time is hasty, the function and the code are relatively primitive, later has the time to improve again. Like to give a star on GitHub. Thank!

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.