The Golang coding paradigm from NSQ

This is a creation in Article, where the information may have evolved or changed.

Managing Goroutines

It ' s surprisingly easy to start goroutines. Unfortunately, it isn ' t quite as easy to orchestrate their cleanup. Avoiding deadlocks is also challenging. Most often this boils a ordering problem, where a goroutine receiving on a Go-chan exits before the Upst Ream Goroutines sending on it.

Why care for all though? It's simple, an orphaned goroutine is a memory leak. Memory leaks in long running daemons was bad, especially when the expectation was that your process would be stable if all else fails.

To further complicate things, a typical NSQD process have many goroutines involved in message delivery. Internally, message "ownership" changes often. To being able to shutdown cleanly, it's incredibly important to account for all intraprocess messages.

Although there aren ' t any magic bullets, the following techniques do it a little easier to manage ...

Waitgroups

syncThe package provides sync.WaitGroup , which can is used to perform accounting of how many Goroutines is live (and provide a M EANs to wait on their exit).

To reduce the typical boilerplate, NSQD uses this wrapper:

type WaitGroupWrapper struct {sync.WaitGroup}func (w *WaitGroupWrapper) Wrap(cb func()) {w.Add(1)go func() {cb()w.Done()}()}// can be used as follows:wg := WaitGroupWrapper{}wg.Wrap(func() { n.idPump() })...wg.Wait()

Exit Signaling

The easiest-trigger a event in multiple child goroutines are to provide a single go-chan so you close when ready. All pending receives on that Go-chan would activate, rather than have to send a separate signal to each goroutine.

func work() {    exitChan := make(chan int)    go task1(exitChan)    go task2(exitChan)    time.Sleep(5 * time.Second)    close(exitChan)}func task1(exitChan chan int) {    <-exitChan    log.Printf("task1 exiting")}func task2(exitChan chan int) {    <-exitChan    log.Printf("task2 exiting")}

Synchronizing Exit

It is quite difficult to implement a reliable, deadlock free, exit path, the accounted for all in-flight messages. A Few tips:

1. Ideally the goroutine responsible for sending on a Go-chan should also is responsible for closing it.

2. If messages cannot is lost, ensure that pertinent go-chans is emptied (especially unbuffered ones!) to guarantee senders can make progress.

3. Alternatively, if a message is no longer relevant, sends on a single Go-chan should being converted to a with the select Addit Ion of an exit signal (as discussed above) to guarantee progress.

4. The general order should is:

   1. Stop Accepting new connections (close listeners)
   2. Signal Exit to Child goroutines (see above)
   3. Wait on for WaitGroup Goroutine exit (see above)
   4. Recover Buffered data
   5. Flush anything left to disk

Logging

Finally, the most important tool at your disposal are to log the entrance and exit of your goroutines!. It makes it infinitely easier to identify the culprit in the case of deadlocks or leaks.

NSQD log lines include information to correlate Goroutines with their siblings (and parent), such as the client ' s remote A Ddress or the Topic/channel name.

The logs is verbose and not verbose to the point where the-the-log is overwhelming. There ' s a fine line, but Nsqd leans towards the side of have more information in the logs when a fault occurs R Ather than trying to reduce chattiness at the expense of usefulness.