Application of Go in codoy database middleware and go codoy Middleware

Application of Go in codoy database middleware and go codoy Middleware

Author: Liu Yanyun

This article mainly focuses on Go's database middleware application. First, we will talk about the application development requirements. For details, we will refer to whether Go's features meet these requirements; next, we will introduce the mysql middleware project recently developed using the Go language. The overall solution is table sharding routing, failover, smooth resizing, and system O & M, mainly from the above five aspects.

Program Development Requirements

The requirements for development languages for program development are summarized as follows:

Refined language features, easy to get started

High development efficiency and clear code Logic

High running performance, saving machine Resources

Easy deployment and maintenance

Comprehensive Ecosystem

Golang features

Based on the differences between Golang and C, the characteristics of Golang are described as follows:

Concise Go syntax; no learning pressure

High development efficiency; the language description capability is close to that of the script language

High performance; close to C/C ++, making full use of resources

Easy to deploy; executable program, which solves dependencies during online deployment and runtime during compilation

Powerful standard library, rich third-party library, go test, pprof

Automatic Memory Management; Memory leakage and wild pointers are the nightmare of C/C ++ developers

Go routine + channel; simple concurrency and simple data synchronization

Go for mysql middleware development

• Overall system solution

Figure 1

The starting point of this system development is to break through the standalone and single table capacity of mysql and solve the single point of access for mysql.

Is the overall framework of the system. The entire system is dedicated to providing a mysql Distributed Solution. Upper-layer applications access the system like standalone mysql; the system performs routing distribution, failover, read/write splitting, and so on.

Proxy receives SQL requests, parses SQL statements, route distribution, and assembles returned results;

Mysql-group is a "Replica set", which can be in master-slave mode or master-master mode;

Dbmoniter is mainly responsible for failover and data repair;

Zookeeper stores configuration information.

• Table sharding routing Logic

Table sharding rules follow two types: Hash Table sharding and segmentation table sharding. Hash Table sharding uses the Hash function to split tables. Hash Table sharding is performed by year, month, day, or Integer Range. The difference between the two methods lies in the differences in the Data Organization Mode due to different hash rules.

• Failover

MHA is a popular solution in the industry for MySQL high availability solutions. MHA is still a little difficult to deploy. You need to deploy an agent on each machine and then perform SSH authorization between machines. Our policy is to configure Rsync to pull mysql binlog.

Fault situations and corresponding solutions:

We have implemented the following master-slave switchover mechanism in dual-master mode, which is not supported by MHA.

Figure 2

Master/Slave switchover and data recovery process: when the Master node fails, it tries to pull the Binlog through Rsync to maximize data loss. The data difference between Slave and Slave is restored through the relay log.

Figure 3

• Online smooth resizing

There are two data migration methods:

How expansion works: mysqldump export inventory data + binlog Increment

Expansion workflow:

First, export existing data;

Second, subscribe to binlog changes and follow up incremental data;

Again, after synchronization, modify the routing rules;

Finally, clear unnecessary redundant data.

The benefit of this design is that any problem occurs in any part of the design and the rollback can be performed immediately, which is relatively safe for data operations. It is the flowchart during resizing.

Figure 4

• System Management commands

Figure 5

5 shows several major management commands, such as online/offline MySQL, data migration commands, and route adding.

End.

Reprinted, please indicate 36 big data (36dsj.com): 36 Big Data» application of Go in codoon database Middleware