Graceful server restart using the Go language

This is a creation in Article, where the information may have evolved or changed.

English Original: Graceful server restart with Go

Go is designed as a background language, and it is often used in back-end programs. The service-side program is the most common software product in the Go language. The question I'm going to address here is: How to cleanly upgrade a running server-side program.

Goal:

• Do not close existing connections: for example, we do not want to shut down a running program that has already been deployed. However, you want to upgrade your service without any restrictions.
• The socket connection responds to user requests at any time: the closing of the socket at any moment may cause the user to return a ' connection denied ' message, which is undesirable.
• The new process needs to be able to start and replace the old ones.

Principle

In Unix-based operating systems, signal (signaling) is a common method of interacting with long-running processes.

• SIGTERM: gracefully Stop the process
• SIGHUP: Reboot/Reload Process (ex: Nginx, sshd, Apache)

If you receive a sighup signal, gracefully restarting the process requires the following steps:

1. The server rejects the new connection request, but maintains the existing connection.
2. To enable a new version of a process
3. "Give" the socket to the new process, and the new process will start accepting new connection requests
4. Stops immediately after the old process has finished processing.

Stop accepting connection requests

Common denominator of server programs: Hold a dead loop to accept connection requests:

for {    conn, err := listener.Accept()    // Handle connection}

The simplest way to jump out of this loop is to set a timeout on the socket listener, and when called listener.SetTimeout(time.Now()) , listener.Accept() it will immediately return one timeout err that you can capture and process:

for {    conn, err := listener.Accept()    if err != nil {        if nerr, ok := err.(net.Err); ok && nerr.Timeout() {            fmt.Println("Stop accepting connections")            return        }    }}

Note that this operation differs from closing listener. This process is still listening on the server port, but the connection request is queued by the operating system's network stack, waiting for a process to accept them.

Start a new process

Go provides a primitive type ForkExec to generate a new process. You can share certain messages, such as file descriptors or environment parameters, with this new process.

execSpec := &syscall.ProcAttr{  Env:   os.Environ(),  Files: []uintptr{os.Stdin.Fd(), os.Stdout.Fd(), os.Stderr.Fd()},}fork, err := syscall.ForkExec(os.Args[0], os.Args, execSpec)[…]

You will find that this process uses exactly the same parameters as the OS. Args initiates a new process.

Send socket to child process and restore it

As you saw earlier, you can pass the file descriptor to the new process, which requires some UNIX magic (everything is a file), and we can send the socket to a new process so that the new process can use it and receive and wait for a new connection.

But the fork-execed process needs to know that it has to get a socket from the file instead of a new one (some may already be in use because we haven't disconnected the existing listener). You can follow any method you want, most commonly through environment variables or command line flags.

listenerFile, err := listener.File()if err != nil {    log.Fatalln("Fail to get socket file descriptor:", err)}listenerFd := listenerFile.Fd()// Set a flag for the new process startprocessos.Setenv("_GRACEFUL_RESTART", "true")execSpec := &syscall.ProcAttr{    Env:   os.Environ(),    Files: []uintptr{os.Stdin.Fd(), os.Stdout.Fd(), os.Stderr.Fd(), listenerFd},}// Fork exec the new version of yourserverfork, err := syscall.ForkExec(os.Args[0], os.Args, execSpec)

Then at the beginning of the program:

var listener *net.TCPListenerif os.Getenv("_GRACEFUL_RESTART") == "true" {    // The second argument should be the filename of the file descriptor    // however, a socker is not a named file but we should fit the interface    // of the os.NewFile function.      file := os.NewFile(3, "")      listener, err := net.FileListener(file)      if err != nil {        // handle      }      var bool ok      listener, ok = listener.(*net.TCPListener)      if !ok {        // handle      }} else {    listener, err = newListenerWithPort(12345)}

The file description is not randomly selected as 3 because the slice of uintptr has been sent fork, and the listener gets the index 3. Beware of implicit declarative issues

Last step, wait for old service connection to stop

So that's it, we've passed it to another process that is running correctly, and the last action for the old server is to wait for its connection to close. Because sync is available in the standard library. Waitgroup structure, using go to achieve this function is very simple.

Each time you receive a connection, add 1 to the Waitgroup, and then we subtract one of the counters when it is complete:

For {conn, err: = Listener. Accept ()

wg.Add(1)    go func() {    handle(conn)    wg.Done()}()

}

As for the end of the waiting connection, you only need WG. Wait (), because there is no new connection, we wait for the WG. Done () has been called by all running handler.

Bonus: Do not wait indefinitely, given a limited time

There is time. Timer, the implementation is simple:

timeout := time.NewTimer(time.Minute)wait := make(chan struct{})go func() {    wg.Wait()    wait <- struct{}{}}()select {case <-timeout.C:    return WaitTimeoutErrorcase <-wait:    return nil}

A complete example

The code snippets in this article are extracted from this complete example: https://github.com/Scalingo/go-graceful-restart-example

Conclusion

The socket delivery Mate Forkexec is really an effective way to do a non-disruptive update process, and the new connection will wait for a few milliseconds at the maximum time-for service startup and recovery sockets, but this is a short time.

This article is part of my # Friday Tech series, and there will be no new updates coming up this week, everybody Merry Christmas.

Link:

• Example: Https://github.com/Scalingo/go-graceful-restart-example

• Go variable Stealth declaration: http://www.qureet.com/blog/golang-beartrap/

• Bonus question: is the fork () Golang process secure? : Https://groups.google.com/forum/#!msg/golang-nuts/beKfn7rujNg/zwY6zwl7QtQJ

• Our website: https://appsdeck.eu

-léo Unbekandt CTO @ appsdeck

All translations in this article are for learning and communication purposes only, please be sure to indicate the translator, source, and link to this article.

Our translation work in accordance with the CC agreement, if our work has violated your rights and interests, please contact us promptly