Query service of IP address information based on Golang

This is a creation in Article, where the information may have evolved or changed.

At work, there is often a need to match user information through IP, such as determining the region in which the user resides (country/province/city), operator, time zone, latitude and longitude, and so on. A while ago there was a Golang development project also has such a demand, so a simple implementation of a package, the recent sneak up on a support HTTP and GRPC mode call service, and open source on GitHub. This article mainly introduces the implementation details and usage methods of IP address information query.

Let's start by explaining the GitHub address:

• Ipquery Golang Package:https://github.com/tabalt/ipquery
• IP Address Information Enquiry service: Https://github.com/tabalt/ipqueryd

Welcome you to use in the project (has passed the test of N billion day PV service), have any questions or suggestions, please submit issue feedback or fork to the name of the modified after submitting pull Request.

IP data files

IP data file storing IP address segment and data information mapping relationship is the most important part of IP address information query, the format requirements can be extended, the data need accurate or even accurate. The real perfect IP data file does not exist, and to keep the data files available, you need to regularly maintain updates to the data files.

Data format

IP data files are usually in plain text form, mapping relationships are stored in rows, each line is separated by "\ T", the first two columns are IP segment start and end points converted to unsigned 32-bit integer value (only consider IPV4), the latter part is the information, can be expanded according to actual needs The requirements between the lines are arranged by IP segment from small to large ascending order. The sample format is as follows:

10332241921033228287北京北京朝阳联通10332282881033232383北京北京海淀联通10332323841033233407北京北京昌平联通

Data sources

There are 3 main sources of IP information data:

• Companies with a certain size will maintain an IP library themselves, and if you work in these companies, you can use them directly.
• There are some free IP libraries on the network (such as the Pure IP library)
• Purchase of commercial IP libraries (e.g. ipip.net)

In addition, we can obtain the maintenance IP information database by some technical or artificial means, but the cost will be very high.

The principle of IP address information query

With the data file, to implement the information query is not difficult, the simple way is to load the data file into an array of memory, the IP address is converted to an unsigned 32-bit integer lookup, and then use the binary lookup method to find the interval of the integral type, and then return the corresponding data, not found to return the failure. The core code in Golang is as follows:

Turn the IP address string into an unsigned 32-bit integer:

func ip2Long(ip string) uint32 {var long uint32binary.Read(bytes.NewBuffer(net.ParseIP(ip).To4()), binary.BigEndian, &long)return long}

Main structure:

type IpRange struct {Begin uint32End   uint32Data  []byte}type IpData []*IpRange

Two-point search:

func (id *IpData) getIpRange(ip string) (*IpRange, error) {var low, high int = 0, (id.Length() - 1)ipdt := *idil := ip2Long(ip)if il <= 0 {return nil, ErrorIpRangeNotFound}for low <= high {var middle int = (high-low)/2 + lowir := ipdt[middle]if il >= ir.Begin && il <= ir.End {return ir, nil} else if il < ir.Begin {high = middle - 1} else {low = middle + 1}}return nil, ErrorIpRangeNotFound}

Golang Ipquery Package Introduction

Usage of Ipquery Package

Ipquery Package (https://github.com/tabalt/ipquery/) is very simple to use, after importing the package through ipquery.Load() method initialization to load the IP data file, and then you can use ipquery.Find() the method to query the IP address corresponding information. The sample code is as follows:

package mainimport (    "fmt"    "github.com/tabalt/ipquery")func main() {    df := "testdata/test_10000.data"    err := ipquery.Load(df)    if err != nil {        fmt.Println(err)    }    ip := "61.149.208.1"    dt, err := ipquery.Find(ip)    if err != nil {        fmt.Println(err)    } else {        fmt.Println(ip, string(dt))    }}

If you want to safely update the data file while the program is running, use the method, and you ipquery.ReLoad() ipquery.Length() can get the total number of data bars loaded into memory.

The methods described above are in fact packaged quickly for ease of use, and can be used to gain greater flexibility by directly using ipquery.NewIpData() the IPDATA structure returned by the method. If a custom Io.reader is passed to the load or reload method of the Ipdata struct, the Ipquery package can be initialized from a non-text file's data source.

Test results of Ipquery package

Ipquery Package provides a more complete unit test, cloning code into the Gopath, enter the $gopath/ipqeury directory, execute the go test relevant command to execute the test code:

[tabalt@localhost ipquery] go test -v=== RUN   TestIpData_Load--- PASS: TestIpData_Load (0.01s)=== RUN   TestIpData_Find--- PASS: TestIpData_Find (0.01s)=== RUN   TestIpData_Parallel_Find--- PASS: TestIpData_Parallel_Find (0.01s)PASSok      ipquery 0.051s

The performance of the Ipquery package from the results of the test is quite good, in a 2-core 4G CentOS 6.2 Golang 1.7.1 Virtual machine Development machine, the initialization of 23M data files on average time-consuming 500ms or so, the implementation of the average time to find 0.012ms, specific data as follows:

[tabalt@localhost ipquery] go test -bench=.BenchmarkIpData_Load-2                 3         452223279 ns/op        97439626 B/op    1780052 allocs/opBenchmarkIpData_Find-2            100000             11472 ns/op            1118 B/op         21 allocs/opPASSok      ipquery 33.488s[tabalt@localhost ipquery] go test -bench=.BenchmarkIpData_Load-2                 3         500309108 ns/op        97439621 B/op    1780052 allocs/opBenchmarkIpData_Find-2            100000             11809 ns/op            1118 B/op         21 allocs/opPASSok      ipquery 33.498s[tabalt@localhost ipquery] go test -bench=.BenchmarkIpData_Load-2                 3         436756760 ns/op        97439621 B/op    1780052 allocs/opBenchmarkIpData_Find-2            100000             12574 ns/op            1118 B/op         21 allocs/opPASSok      ipquery 34.510s

IP Address Information Inquiry Service Introduction

As stated at the beginning of the article, this project provides HTTP and GRPC interfaces based on the Ipquery package, and the name is very vulgar to take as ipqueryd. Personal habits The project-level go code is not placed in the global Gopath, but instead uses the shell script to dynamically modify Gopath to execute the GO command after the project directory, so you can use the following steps to run the project:

[tabalt@localhost ~] git clone https://github.com/tabalt/ipqueryd.git ~/$NOT_YOUR_GOPATH/[tabalt@localhost ipqueryd] cd ~/$NOT_YOUR_GOPATH/ipqueryd[tabalt@localhost ipqueryd] ./ctrl.sh run

Configuration file

Conf directory in the project has a Ipqueryd.json configuration file, you can configure the PID file, HTTP service port, GRPC service port, data file path and other content, can be modified according to requirements; The service port can be either one or two.

{    "pid_file": "./tmp/ipqueryd.pid",    "http_server_port": ":12101",    "grpc_server_port": ":12102",    "data_file": "./data/ip_data.txt"}

HTTP interface

The HTTP interface supports the return of JSON format and JSONP format responses, which are tested using the command line:

[tabalt@localhost ipqueryd] curl "http://127.0.0.1:12101/find?ip=1.1.8.1"{"data":["广东省电信"]}[tabalt@localhost ipqueryd] curl "http://127.0.0.1:12101/find?ip=1.1.8.1&_callback=showip"showip({"data":["广东省电信"]});

Grpc interface

The Grpc interface needs to be written in the language you are familiar with, and the following code is simple to use in Golang:

package mainimport ("log""golang.org/x/net/context""google.golang.org/grpc""pb")func main() {conn, err := grpc.Dial("127.0.0.1:12102", grpc.WithInsecure())if err != nil {log.Fatalf("did not connect: %v", err)}defer conn.Close()iqc := pb.NewIpQueryClient(conn)ip := "1.1.8.1"r, err := iqc.Find(context.Background(), &pb.IpFindRequest{Ip: ip})if err != nil {log.Fatalf("could not find: %v", err)}log.Printf("ip data: %s", r.Data)}

More content waiting for you to discover and contribute!

This article link: http://tabalt.net/blog/ipquery-server-by-golang/, reprint please specify.