Implementing a proxy pool with Golang

This is a creation in Article, where the information may have evolved or changed.

Background

When writing a crawler, you will always encounter the speed of crawling too fast and be blocked by IP, this time you need to use the agent. In Https://github.com/henson/ProxyPool
Inspired, decided to implement a proxy pool. Project has been open source on GitHub.

Https://github.com/AceDarkknight/GoProxyCollector

Development environment

Windows 7,go 1.8.4

Data sources

Http://www.xicidaili.com
http://www.89ip.cn
http://www.kxdaili.com/
Https://www.kuaidaili.com
http://www.ip3366.net/
http://www.ip181.com/
Http://www.data5u.com
Https://proxy.coderbusy.com

Project structure

Catalogue	function
Collector	Collectors, crawling agents for each website
Result	Represents the result of a crawl
Scheduler	Responsible for task scheduling, including starting collector and warehousing
Server	Start a Web service that provides a result-fetching API
Storage	Store results, other databases can be used through interfaces
Util	Some common tools and methods
Verifier	Authentication of IP and inbound out of library

Realize

• Collector

  Collector supports two modes, using Goquery to select a page element and use regular expressions to match the information we need. Go directly to the code.

Github.com\acedarkknight\goproxycollector\collector\selectorcollector.gofunc (c *SelectorCollector) Collect (ch chan<-*result.    Result) {//close channel before exiting. Defer close (CH) Response, _, errs: = Gorequest. New (). Get (C.currenturl). Set ("User-agent", util. Randomua ()).    End ()/* Omit part of the code *////Some sites are not UTF-8 encoded and need to be transcoded. var decoder mahonia. Decoder if C.configuration.charset! = "Utf-8" {Decoder = Mahonia.    Newdecoder (C.configuration.charset)}//Use Goquery. Doc, Err: = Goquery. Newdocumentfromreader (response. Body) If err! = Nil {seelog.    Errorf ("Parse%s error:%v", C.currenturl, Err) return}//the proxy list for most proxy sites is placed in a table and the elements in the table re-loop are selected first. Selection: = Doc. Find (c.selectormap["table"][0]) selection. Each (func (i int, sel *goquery. Selection) {var (IP string port int speed Float64 Locati        on string)//the name and path of the information we need exists collectorconfig.xml. Namevalue: = Make (map[string]string) for key, value: = Range C.selectormap {if key! = "Table" {var temp s Tring If len (value) = = 1 {temp = sel. Find (Value[0]). Text ()} else if Len (value) = = 2 {temp, _ = sel. Find (Value[0]).                Attr (value[1])}//transcoding. If temp! = "" {if decoder! = Nil {temp = decoder.        Convertstring (temp)} Namevalue[key] = temp}}} /* Omit part of the code *//filter for some non-qualifying results if IP! = "" && port > 0 && Speed >= 0 && sp Eed < 3 {r: = &result. result{Ip:ip, Port:port, Location:location, Spee D:speed, Source:c.currenturl}//Put the eligible results in Channel CH <-r}})}/ /Github.com\acedarkknight\goproxycollector\collector\regexcollector.gofunc (c *regexcollector) Collect (Ch chan<- *result. Result) {response, bodystring, errs: = Gorequest. New (). Get (C.currenturl). Set ("User-agent", util. Randomua ()).    End ()/* Omit part of the code *///match with regular. Regex: = RegExp. Mustcompile (c.selectormap["IP"]) ipaddresses: = Regex. Findallstring (bodystring,-1) If Len (ipaddresses) <= 0 {seelog. Errorf ("Can not found correct format IP address in url:%s", C.currenturl) return} for _, ipAddress: = Range ipaddresses {temp: = strings. Split (IpAddress, ":") If Len (temp) = = 2 {port, _: = StrConv. Atoi (temp[1]) if port <= 0 {Continue} r: = &result.            result{Ip:temp[0], Port:port, Source:c.currenturl,} CH <-r}}}

• Result

  Result is simple, just to represent the result of a collector crawl.

// github.com\AceDarkknight\GoProxyCollector\result\result.gotype Result struct {    Ip       string  `json:"ip"`    Port     int     `json:"port"`    Location string  `json:"location,omitempty"`    Source   string  `json:"source"`    Speed    float64 `json:"speed,omitempty"`}

• Scheduler

  Scheduler is responsible for doing some initialization work and dispatching collector tasks. Different tasks run in different goroutine, and Goroutine communicate through the channel.

Github.com\acedarkknight\goproxycollector\scheduler\scheduler.gofunc Run (configs *collector. Configs, storage storage. Storage) {/* Omit part of the code */for {var wg sync. Waitgroup for _, Configuration: = Range configs. CONFIGS {WG. ADD (1) Go func (c collector.                Config) {//Prevent deadlocks. Defer WG.                Done ()//handling panic. Defer func () {if r: = Recover (); r! = nil {seelog. Criticalf ("Collector%s occur panic%v", C.name, R)}} () Col: = C.collec                    Tor () Done: = Make (chan bool, 1) go func () {runcollector (col, storage)                    Signal is sent when complete.                Done <-True} ()//Set timeout to prevent Goroutine from running too long. Select {Case <-done:seelog.              DEBUGF ("Collector%s finish.", C.name)  Case <-time. After (7 * time. Minute): Seelog.         Errorf ("Collector%s time out.", C.name)}} (Configuration)}//waits for all collector to complete. Wg. Wait () Seelog.        Debug ("Finish once, sleep minutes.") Time. Sleep (time. Minute * 10)}}

• Server

  Server has launched an API

• Storage

  Storage provides storage-related interface and implementations.

// github.com\AceDarkknight\GoProxyCollector\storage\storage.gotype Storage interface {    Exist(string) bool    Get(string) []byte    Delete(string) bool    AddOrUpdate(string, interface{}) error    GetAll() map[string][]byte    Close()    GetRandomOne() (string, []byte)}

The current project data are stored in BOLTDB. GitHub's introduction to Boltdb is as follows:

Bolt is a pure Go Key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable database for projects that don ' t require a full databas E server such as Postgres or MySQL.
Since Bolt is meant to being used as such a low-level piece of functionality, simplicity is key. The API is small and only focus on getting values and setting values. That ' s it.

Given the small amount of data in the agent pool, and the idea is to implement an out-of-the-box proxy pool, choosing an embedded database such as BOLTDB is clearly simpler and more convenient than using MySQL and MongoDB. Of course, if you want to use a different database later, you only need to implement the storage interface. The relevant documentation and tutorials for using boltdb are in my reference:

https://segmentfault.com/a/1190000010098668

Https://godoc.org/github.com/boltdb/bolt

• Util

  Util implements a number of common methods, such as taking a random user-agent, which is not expanded.

• Verifier

  Verifier is responsible for verifying that collector gets the IP is available, the available storage, is not available to remove from the database.

  Configuration

  The collector is driven by a configuration file. The configuration files are:

github.com\AceDarkknight\GoProxyCollector\collectorConfig.xml

As an example:

<configname="Coderbusy">    <urlFormat>https://proxy.coderbusy.com/classical/https-ready.aspx?page=%s</urlFormat>    <urlParameters>The</urlParameters>    <collectType>0</collectType>    <charset>Utf-8</charset>    <valueNameRuleMap>        <itemname="Table"rule=". Table Tr:not (: first-child)"/>        <itemname="IP"rule="Td:nth-child (2)"attribute="Data-ip"/>        <itemname="Port"rule=". Port-box"/>        <itemname="Location"rule="Td:nth-child (3)"/>        <itemname="Speed"rule="Td:nth-child (Ten)"/>    </valueNameRuleMap></config><configname="89ip">    <urlFormat>http://www.89ip.cn/tiqv.php?sxb=&amp;Tqsl=20&amp;ports=&amp;ktip=&amp;Xl=on&amp;Submit=%cc%e1++%c8%a1</urlFormat>    <collectType>1</collectType>    <charset>Utf-8</charset>    <valueNameRuleMap>        <itemname="IP"rule="((?:(? : 25[0-5]|2[0-4]\d| ((1\d{2}) | ([1-9]?\d))) \.) {3} (?: 25[0-5]|2[0-4]\d| ((1\d{2}) | ([1-9]?\d))): [1-9]\d*]/>    </valueNameRuleMap></config>

• Name is the name of collector, the main function is to facilitate debugging and error-checking problems.

• Urlformat and urlparameters are used to stitch up URLs that need to be crawled. The urlparameters can be empty. For example, the first configuration above is to tell the crawler to crawl the site is:

  Https://proxy.coderbusy.com/classical/https-ready.aspx?page=1

  https://proxy.coderbusy.com/classical/https-ready.aspx?page=2

• Collecttype indicates which collector,0 is used to represent selectorcollector,1 on behalf of Regexcollector.
• CharSet indicates what kind of encoding the site uses. The default encoding is UTF-8, and you may not get the data you want if you set it wrong.

• Valuenamerulemap the rule that represents the desired point. For sites that use Selectorcollector, most of the results are represented by table, so table is required, and other points are configured according to different Web sites. The configuration of the relevant rule can refer to the Goquery documentation:

  Https://github.com/PuerkitoBio/goquery

Conclusion

About the introduction of the project is almost here, the first time a novice to write a project with go if there are any shortcomings and errors hope that you can forgive and point out. If you have questions and better suggestions, you are welcome to explore together ~