Example analysis of performance optimization of Go Language Project (Kingshard)

This is a creation in Article, where the information may have evolved or changed.

Kingshard Performance Optimization Network Chapter

Recently, Kingshard has slowed down a lot in its function development rhythm. On the one hand, the work is really busy, on the other hand, I think the function of Kingshard has been more perfect, the next development focus should be performance optimization. After all, as a MySQL proxy, if the performance of the forwarding SQL is very poor, no matter how many functions are useless. So this weekend has been home to optimize the forwarding performance of Kingshard. After two days of discovery, the Kingshard's forwarding SQL performance was improved by about 18%, and learned a bit about it in this process. Take this opportunity to share, but also to urge yourself to write blog enthusiasm. ：）

1. Discovery of Kingshard Performance bottlenecks

First choice, for Kingshard performance optimizations, we have to find out where Kingshard's performance bottlenecks are. The go language is very good at performance optimization support, with the help of the Go language Pprof tool, we can get the time-consuming function of Kingshard when forwarding SQL requests through a few simple steps.

1.1 Environment Construction

Build a Kingshard agent environment based on the Kingshard usage guidelines. I am a MacBook-built environment, the hardware parameters are as follows:

CPU： 2.2GHZ * 4内存：16GB硬盘: 256GB

1.2 Performance test Steps

The specific steps are as follows:

1. Get a package Library of profiling

go get github.com/davecheney/profile

2. Import this component within the project

3. Add the start and stop portals for CPU monitoring at the beginning of the main function of Kingshard/cmd/kingshard/main.go

func main() {    defer profile.Start(profile.CPUProfile).Stop()    fmt.Print(banner)    runtime.GOMAXPROCS(runtime.NumCPU())    flag.Parse()    ....}

4. Recompile the project, run Kingshard

./bin/kingshard -config=etc/ks.yaml

After the 5.kingshard starts, it will output the following paragraph in the terminal:

2015/10/31 10:28:06 profile: cpu profiling enabled, /var/folders/4q/zzb55sfj377b6vdyz2brt6sc0000gn/T/profile205276958/cpu.pprof

The path behind is the location of the pprof profiling file, and CTRL + C interrupts the server

6. This time with Sysbench to kingshard pressure test, get QPS (about Sysbench installation and use, please Google solve). The specific code is as follows:

sysbench --test=oltp --num-threads=16 --max-requests=160000 --oltp-test-mode=nontrx --db-driver=mysql --mysql-db=kingshard --mysql-host=127.0.0.1 --mysql-port=9696 --mysql-table-engine=innodb --oltp-table-size=10000 --mysql-user=kingshard --mysql-password=kingshard --oltp-nontrx-mode=select --db-ps-mode=disable run

The following results are obtained:

OLTP Test statistics:queries performed:read:160071 write:                        0 other:0 total:160071 transactions:    160071 (16552.58 per sec.)    Deadlocks:0 (0.00 per Sec.)    Read/write requests:160071 (16552.58 per sec.) Other operations:0 (0.00 per Sec.)    Test execution Summary:total time:9.6705s total number of events:160071 Total time taken by event execution:154.4474 Per-request statistics:min:0.2 9ms avg:0.96ms max:14.17ms appr  Ox. Percentile:1.37msthreads fairness:events (Avg/stddev): 10004.4375/24.95 Execution time (Avg/stddev): 9.6530/0.00 

• Follow the above steps to test the average of three (16552.58,16769.72,16550.16), the Kingshard is optimized before the QPS is: 16624.15

• Follow the steps above and connect to MySQL directly. Test directly with MySQL QPS, also test three times QPS (27730.90,28499.05,27119.20), get straight to MySQL's QPS is: 27783.05.

• From the above data can calculate the performance of Kingshard forwarding SQL is directly connected to MySQL about 59%.

7. Copy the cpu.prof to the Bin/kingshard location

8. Call the Go tool to create a CPU-time-consuming PDF document

go tool pprof -pdf ./kingshard cpu.pprof > report.pdf

2. Performance Test Report Analysis

The above command allows you to generate the time-consuming situation of the main function during the pressure test. From the report point of view, the main time-consuming is the TCP layer packet send and receive above. Then we should mainly consider how to optimize the TCP layer data transmission and delivery aspects. To optimize TCP transfer efficiency, I first thought about reducing system calls, transmitting as much data as possible to each packet.

When communicating over a TCP socket, the data is split into chunks so that they can be encapsulated in a given connection's TCP payload (referred to as payload in TCP packets). The size of the TCP payload depends on several factors, such as the maximum message length and path, but these factors are known when the connection is initiated. To achieve the best performance, our goal is to populate each message with as much data as available. When there is not enough data to fill payload (also known as the maximum segment length (maximum segment size) or MSS), TCP uses the Nagle algorithm to automatically connect some small buffers to a segment of the message. This can improve the efficiency of the application by minimizing the number of messages sent, and alleviate the overall network congestion problem.

Because this algorithm merges the data, it attempts to form a complete TCP segment, so it introduces some delay. However, this algorithm minimizes the number of packets sent on the line, thus minimizing the problem of network congestion. However, in the case of minimizing transmission delays, the sockets API in the Go language provides a solution. Is through:

func (c *TCPConn) SetNoDelay(noDelay bool) error

This function is set to True by default in go, that is, the delay option is not turned on. We need to set it to false to get as much data per packet as possible, reducing the purpose of the system call.

2.1 Code modification and performance testing

After discovering the performance bottleneck, modify the Newclientconn function and Backend/backend_ in the Proxy/server/server.go file The Reconnect function in Conn.go sets the connection between client and Kingshard and Kingshard to MySQL to minimize transmission delay. Specific code changes can be viewed in this commit.

After the modification we re-tested with Sysbench, the test command is consistent with the above test. The results are as follows:

OLTP Test statistics:queries performed:read:160174 write:                        0 other:0 total:160174 transactions:    160174 (21291.68 per sec.)    Deadlocks:0 (0.00 per Sec.)    Read/write requests:160174 (21291.68 per sec.) Other operations:0 (0.00 per Sec.)    Test execution Summary:total time:7.5228s total number of events:160174 Total time taken by event execution:119.9655 Per-request statistics:min:0.2 6ms avg:0.75ms max:10.78ms appr  Ox. Percentile:1.13msthreads fairness:events (Avg/stddev): 10010.8750/38.65 Execution time (Avg/stddev): 7.4978/0.00 

The QPS that was tested three times was: 21291.68,21670.85,21463.44. equivalent to about 77% of the direct-attached MySQL performance, this optimization improves performance by around 18% .

Summarize

In this article, the detailed steps for performance analysis of Kingshard are presented through the pprof provided by the go language. For other go language projects, you can also generate performance report documents with similar steps. The key to performance optimization is to find the performance bottleneck and then find the optimization solution. Sometimes the simple optimization, can achieve the unexpected effect, I hope this article can give go developers in performance optimization to provide a way of thinking. Last ad: Kingshard as an open source MySQL middleware project that supports sharding, the performance of the forwarding SQL has improved a lot after the performance is optimized. I will also optimize the kingshard in terms of lock and memory, so please look forward to it.

Github:https://github.com/flike/kingshard