Chapter II Commissioning of Goroutine leaks

This is a creation in Article, where the information may have evolved or changed.

Before we talk about the Goroutines leak, let's look at the concept of concurrent programming. Concurrent execution of concurrent programming handlers. Multiple continuous flow tasks are executed concurrently through concurrent programming, resulting in faster execution. For modern software running on multicore processors, concurrent programming is necessary to make better use of the capabilities of multi-core processors for faster concurrent/parallel programs.

Co-process (Goroutines)

The implementation of the concurrent execution, the process is the go run-time lightweight thread, the process and the thread does not have a one-to-one relationship, the process is scheduled by the go management, run on different threads. The go process is designed to hide many of the complexities involved in threading creation and management.

With respect to concurrent/parallel programs, concurrent programs may or may not be parallel. Parallelism is the ability to increase speed by using multiple processors. A well-designed concurrency program is also performing well in terms of parallelism. In the go language, in order for your program to run with multiple cores, the process is actually running in parallel, and you must use the Gomaxprocs variable. Detailed reference: HTTPS://GITHUB.COM/UNKNWON/THE-WAY-TO-GO_ZH_CN/BLOB/MASTER/EBOOK/14.1.MD

Sync (Synchronize)

Processes, threads, and co-worker collaboration have a common goal: synchronization and communication.

In the go language, channels is used for synchronization of the co-processes. Traditional threading mode communication is shared memory. Go encourages the use of the channel to pass references between the threads, rather than explicitly using locks to coordinate access to shared data. This approach ensures that only one goroutine can access the data at a given time.

As shown in the following example, after each worker executes, they need to work with the main coprocessor to pass the returned results through channels to the main coprocessor, after which the main thread exits the program.

Synchronization error

Notice how the Go routine exits each time you use the GO keyword. Sometimes syncing can cause errors, causing some goroutine to wait forever. In the Go language, the following conditions can cause synchronization errors:

Channel not accepted by

There is no recipient to accept the data sent by the sender, the channel is blocked. A channel with no recipient can cause the program to hang. In the following example, CH1 has no recipient and will cause the channel to be blocked.

Package Main

Import "FMT"

Func main() {

CH1: =make (Chanint)

Go pump (CH1)//Pump hangs

Fmt. PRINTLN (<-CH1)//prints only 0

}

Funcpump(chchanint) {

Fori: = 0;; i++ {

CH <-I

}

}

Channel not written by

There is a case where the channel does not have a writer and a goroutine leak occurs.

Example 1:for-select

for {

Select {

Case <-C:

Process here

}

}

Example 2:channel cycle

Go func () {

For range Ch {}

}()

Example 3: Demonstrating the tasks loop, which causes the channel to have no writers, requires the main program to call Close (tasks) to avoid goroutine leaks.

Package Main

Import "FMT"

Func concurrency () {

Lets first create a channel with a buffer

tasks: = Make (Chan string, 20)

Create another one to receive the results

Results: = Make (Chan string, 20)

Workers: = []int{1, 2, 3, 4}

Inserting tasks inside the channel

For task: = 0; Task < 10; task++ {

Tasks <-FMT. Sprintf ("Task%d", task)

}

For _, W: = Range Workers {

Starging one goroutine for each worker

Go Work (w, tasks, results)

}

Close (tasks)

Lets print the Resutls

Fmt. Println ("Would print the results")

For res: = 0; Res < 10; res++ {

Fmt. Println ("Result:", <-results)

}

}

Func work (Workerid int, Tasks Chan string, results Chan string) {

Worker would block util a new task arrives in the channel

For T: = range Tasks {

Simple task as Example

Results <-FMT. Sprintf ("Worker%d got%v", Workerid, T)

}

}

Func Main () {

Concurrency ()

}

Good practice.

Using timeout

Timeout: = Make (chan bool, 1)

Go func () {

Time. Sleep (1E9)//One second

Timeout <-True

}()

Select {

Case <-Ch:

A read from CH have occurred

Case <-Timeout:

The read from CH have timed out

} OR Select {

Case Res: = <-c1:

Fmt. Println (RES)

Case <-time. After (time. Second * 1):

Fmt. PRINTLN ("Timeout 1")

}

Use Golang Context Package

Golang Context package can be used to gracefully end routines and even timeout

Leak detection

Instrument ( instrumentation) Endpoint

The way to detect a Web server leak is to add an instrument endpoint and use it with the load test.

Get the count of number of go routines in the system.

Func countgoroutines () int {

Returnruntime. Numgoroutine ()

}

Func Getgoroutinescounthandler (w http. Responsewriter, R *http. Request) {

Get the count of number of go routines running.

Count: = Countgoroutines ()

W.write ([]byte (StrConv. Itoa (count)))

}

Func Main () {

http. Handlefunc ("/_count", Getgoroutinescounthandler)

}

The number of goroutines that exist in the system through the instrument endpoint before and after the load test. The following is the flow of the load test program:

Step 1:call The instrumentation endpoint and get the count of number of goroutines alive in your webserver.

Step 2:perform load Test. Lets the load be concurrent.

For I: = 0; I < 100; i++ {

Go Callendpointunderinvestigation ()

}

Step 3:call The instrumentation endpoint and get the count of number of goroutines alive in your webserver.

If there is an abnormally increased number of goroutine in the system after the load test, a leak is shown. This is a small example of a Web server with a vulnerable endpoint. With a simple test we can determine if there is a leak on the server.

First run the leaky server $ go run leaky-server.go

Run the load test now.$ go run load.go

3 Go routines before the load test in the system.

The "Go routines after the" load test in the system.

You can clearly see that with 50 concurrent requests to the leak endpoint, 50 programs are added to the system.

Let's run the load test again.

$ go Run load.go

Routines Go before the load test in the system.

104 Go Routines after the load test in the system.

It is clear that in each run of the load test, the number of executions in the server is increasing, not decreasing. This is a clear evidence of leakage.

Identify the cause of the leak

Using stacks to track endpoints

Once a leak is found in the Web server, the source of the leak needs to be identified. You can help identify the source of a leak by adding a stack trace endpoint that returns a Web server.

Import (

"Runtime/debug"

"Runtime/pprof"

)

Func Getstacktracehandler (w http. Responsewriter, R *http. Request) {

Stack: = Debug. Stack ()

W.write (Stack)

Pprof. Lookup ("Goroutine"). WriteTo (W, 2)

}

Func Main () {

http. Handlefunc ("/_stack", Getstacktracehandler)

}

After determining the existence of a leak, use the endpoint to obtain stack trace information before and after the load to identify the source of the leak.

Add the Stack trace tool to the leaking server and perform the load test again.

The following stack trace information clearly points to the epicenter of the leak:

First run the leaky server$ go run leaky-server.go

Run the load test now.$ go run load.go

3 Go routines before the load test in the system.

The "Go routines after the" load test in the system. Goroutine 149 [Chan send]:

Main.sum (0xc420122e58, 0x3, 0x3, 0xc420112240)

/home/karthic/gophercon/count-instrument.go:39 +0x6c

Created by main.sumconcurrent

/home/karthic/gophercon/count-instrument.go:51 +0x12b

Goroutine 243 [Chan send]:

Main.sum (0xc42021a0d8, 0x3, 0x3, 0XC4202760C0)

/home/karthic/gophercon/count-instrument.go:39 +0x6c

Created by main.sumconcurrent

/home/karthic/gophercon/count-instrument.go:51 +0x12b

Goroutine 259 [Chan send]:

Main.sum (0xc4202700d8, 0x3, 0x3, 0XC42029C0C0)

/home/karthic/gophercon/count-instrument.go:39 +0x6c

Created by main.sumconcurrent

/home/karthic/gophercon/count-instrument.go:51 +0x12b

Goroutine 135 [Chan send]:

Main.sum (0xc420226348, 0x3, 0x3, 0xc4202363c0)

/home/karthic/gophercon/count-instrument.go:39 +0x6c

Created by main.sumconcurrent

/home/karthic/gophercon/count-instrument.go:51 +0x12b

Goroutine 166 [Chan send]:

Main.sum (0xc4202482b8, 0x3, 0x3, 0xc42006b8c0)

/home/karthic/gophercon/count-instrument.go:39 +0x6c

Created by main.sumconcurrent

/home/karthic/gophercon/count-instrument.go:51 +0x12b

Goroutine 199 [Chan send]:

Main.sum (0xc420260378, 0x3, 0x3, 0xc420256480)

/home/karthic/gophercon/count-instrument.go:39 +0x6c

Created by main.sumconcurrent

/home/karthic/gophercon/count-instrument.go:51 +0x12b

........

Using profiling

Because leaked goroutine are usually prevented from trying to read or write to the channel or even sleep, profilling analysis will help identify the cause of the leak. See benchmarks and profiling for benchmark testing and analysis, or HTTPS://GITHUB.COM/UNKNWON/THE-WAY-TO-GO_ZH_CN/BLOB/MASTER/EBOOK/13.10.MD.

Avoid leaks, sooner

The use of the instrument mechanism in unit and functional testing can help identify leaks early. Number of goroutine before and after the count test.

Func Testmyfunc () {

Get count of Go routines. Perform the test.

Get the Count diff.

Alert if there ' s an unexpected rise.

}

Stack differences in the test

The stack difference is a simple program that compares stack traces before and after testing and alerts you of any unwanted goroutine legacy systems. Integrate it with unit tests and functional tests to help identify leaks during development.

Import (

Github.com/fortytw2/leaktest

)

Func testmyfunc (t *testing. T) {

Defer leaktest. Check (t) ()

Go func () {

for {

Time. Sleep (time. Second)

}

}()

}

Safety design

When a system is affected by a leak or resource interruption of an endpoint/service, the service of the MicroServices architecture can protect the entire system as a standalone container/process operation. Container orchestration tools such as Kubernetes,mesosphere and Docker Swarm are recommended.

Goroutine leaks are like chronic suicides. Imagine getting a stack trace of the entire system and trying to identify which services are causing leaks in hundreds of services! It's scary!!!! They waste your computing resources for a while, and you don't even notice it. It's really important to realize the leaks and debug them as soon as possible!

Go to make your love programming again. I Promise.

Go will let you love programming again. I promise.

Reference:

The "Go to Go" Chinese version of the Get Started guide Https://github.com/Unknwon/the-way-to-go_ZH_CN

Debugging Go routine leaks:https://youtu.be/hwo0fevr92a

Https://github.com/fortytw2/leaktest

Http://www.tuicool.com/articles/2AZf63J