Morestack and Goroutine Pool

This is a creation in Article, where the information may have evolved or changed.

Morestack and Goroutine Pool

2017-07-21

The Goroutine initial stack size of the go language is only 2 K, and if the call chain is longer during the run, the stack will automatically expand when it exceeds this size. This time it will be called to a function runtime.morestack . Opening a goroutine itself is very small, but the overhead of calling Morestack to expand the stack is relatively large. Think about what happens if the stack of functions is expanded and someone references an object on the original stack? So when the Morestack, the objects in the inside need to adjust the position, the pointer will be relocated to the new stack. The larger the stack, the more objects that need to be adjusted, and the greater the overhead when calling Morestack.

We can write a simple bench, the function recursive call is to compare the consumption of stack space:

func f(n int) {  var useStack [100]byte  if n == 0 {    return  }  _ = useStack[3]  f(n - 1)}

Here is the comparison test:

func bench1() {  var wg sync.WaitGroup  for i := 0; i < benchCount; i++ {    wg.Add(1)    go func() {      for j := 0; j < 15; j++ {        f(2)      }      wg.Done()    }()    wg.Wait()  }}func bench2() {  var wg sync.WaitGroup  for i := 0; i < benchCount; i++ {    wg.Add(1)    go func() {      f(30)      wg.Done()    }()    wg.Wait()  }}

The workload is the same, but Bench1 will not trigger runtime.morestack , BENCH2 will trigger, you can see the result is a difference in order of magnitude:

bench1 used: 52480486 nsbench2 used: 559074503 ns

We found this problem in our project, and Morestack's CPU time accounted for almost 10%. Cockroach, the next-door friend, also found the problem. How to solve it?

There are two directions, one is the initial allocation of a larger initial stack, for example, after the Goroutine, call the following function, the stack spread to 8 K:

// reserveStack reserves 8KB memory on the stack to avoid runtime.morestack.func reserveStack(dummy bool) {  var buf [8 << 10]byte  // avoid compiler optimize the buf out.  if dummy {    for i := range buf {      buf[i] = byte(i)    }  }}

The way to reserve stack space in advance, compared to the program run to the back stack is not enough to spread the stack, lower overhead. Cockroach is this approach. I found that the cost of Morestak was not eliminated, but shifted.

So what I'm going to say is another direction, Goroutine pool.

In fact, goroutine such a light amount of things, in fact, it does not mean that the pool, with the use of the open, after the throw, very good. However, in the case of triggering morestack, this overhead is a bit high and can be caught on the fire map (go Pprof is less sensitive). After using the pool, if the goroutine is expanded stack, and then into the pool, the next time it is a stack of goroutine, so you can avoid morestack.

Now let's talk about how this goroutine pool should be written.

I want the interface to be this child:

pool = New()  // 创建poolpool.Go(func() {    // do something})

followed by the call

go func() {}()

The effect is exactly the same, except that pool.Go after the closure function is executed, the goroutine is not exited but is put back into the pool for the next call.

To do this, we are going to abstract goroutine into a resource,

func (pool *Pool) Go(f func()) {     res := pool.get()     res.run(f)     // pool.put(res) 这里还不能归还，后面讲为什么}

This resource is very special, it is composed of a channel and a background goroutine:

type res struct {     ch chan     pool *Pool}go func(r *res) {   for work := r.ch {       work()       r.pool.put(res)   }}

Run only needs to be fed to the channel, and the backstage goroutine will execute it when it gets work done.

func (r *res) run(f) {    r.ch <- f }

Here is a detail, after the execution of the return to the Goroutine pool, to leave work after the completion of the execution, because res.run(f) of non-blocking.

The realization of the pond is relatively easy, with a linked list can be, the res string up, the end of the knot, the tail into the head out. However, to be aware of thread safety, it is possible to optimize the access of the end-to-end nodes to lock, even with no lock formation.

Notice that I didn't design the close interface for pool, why? This is intentional. So, when does the goroutine in the pool release? Designed for a period of time without automatic recycling.

From experience, there is a great probability that a leak problem will occur as long as the code involved in reusing Goroutine, especially the implementation of close and close is called. Allocated resources, it belongs to the pond, or not belong to the pond? The closed pool is waiting for resources to return? Or does it wait? If the resource is returned, the pool is closed? Close the moment, with the request to read and write the pool, how to handle the lock? So this is a design problem.

Each use of the last use of the time, a lot of a collection of Goroutine periodic scan pool of goroutine, if not used for a long time, will be recycled. This recycled goroutine if found a goroutine in the pool is gone, it also exits, very clean, there is no leakage. Add a marker before exiting, and if the pool is used again, recreate the recycle goroutine to work.

Complete code can look inside this pr.