Graceful HTTP service off in Go

This is a creation in Article, where the information may have evolved or changed.

While it's cool to write a 7x24-hour uninterrupted service, we still have to consider how to gracefully end the service at some point, such as service upgrades, configuration updates, and so on.

Of course, the most violent practice is directly kill -9 , but this direct result is likely to kill a lot of running to half of the task, eventually lead to inconsistent data, the bitter fruit only met the talent can deeply understand, the data repair really very painful, and sometimes have to pay the user.

So, usually we are to send a signal to the service, Sigterm also line, Siginterrupt also become, anyway to let the service know the end. After the service receives the end signal, it first rejects all external new requests, and then waits until all the currently executing requests are completed, at the end. Of course, there is a good chance that a time-consuming task is currently being performed, causing the service to not end for a long time, and then deciding whether to force the end.

specifically to go HTTP server inside, how gracefully end an HTTP server?

First of all, we need to show the creation of a listener, let it loop constantly accept the new connection for server processing, why not use the default HTTP. Listenandserve, the main reason is that we can at the end of the time by shutting down this listener to actively reject the external new connection request. The code is as follows:

l, _ := net.Listen("tcp", address)svr := http.Server{Handler: handler}svr.Serve(l)

Serve this function is a dead loop, and we can end it externally with the corresponding listener of close.

When listener accept to the new request, will open a new goroutine to execute, then at the end of the server, how do we know this goroutine is complete?

In the early days, probably go1.2, the author uses sync Waitgroup at the handler entrance, because we have a unified portal handler, so it's easy to know if the request is complete, such as:

func (h *Handler) ServeHTTP(w ResponseWriter, r *Request) {    h.svr.wg.Add(1)    defer h.svr.wg.Done()    ......}

But this is really just to determine whether the request is over, we know that in HTTP 1.1, connection is able to keepalived, that is, the request processing is complete, but connection is still available, We don't have a good way to close off this connection. But then again, we just need to ensure that the current request to the normal end, connection can normal close really doesn't matter, after all, the service is over, connection automatically close. But who called the author is a typical virgo?

After go1.3, we provided a connstate hook that we could use to get to the corresponding connection so that we could close the connection at the end of the service. The hook will be called in the following connstate states.

• Statenew: New connection and ready to send request now
• Stateactive: Indicates that a connection has received one or more bytes of request data, calling the hook before the server invokes the actual handler.
• Stateidle: Indicates that a connection has been processed to complete a request, but because it is keepalived, it will not close and continue to wait for the next request.
• Statehijacked: Indicates that the external call is hijack, the final state.
• Stateclosed: Indicates that the connection has ended, and the final state.

In general, we do not enter the state of hijacked (if WebSocket is considered), so a possible hook function is as follows, refer to http://rcrowley.org/talks/gophercon-2014.html

s.ConnState = func(conn net.Conn, state http.ConnState) {    switch state {    case http.StateNew:        // 新的连接，计数加1        s.wg.Add(1)    case http.StateActive:        // 有新的请求，从idle conn pool中移除        s.mu.Lock()        delete(s.conns, conn.LocalAddr().String())        s.mu.Unlock()    case http.StateIdle:        select {        case <-s.quit:            // 如果要关闭了，直接Close，否则加入idle conn pool中。            conn.Close()        default:            s.mu.Lock()            s.conns[conn.LocalAddr().String()] = conn            s.mu.Unlock()        }    case http.StateHijacked, http.StateClosed:        // conn已经closed了，计数减一        s.wg.Done()    }

At the end of the day, the following process will follow:

func (s *Server) Close() error {    // close quit channel, 广播我要结束啦    close(s.quit)    // 关闭keepalived，请求返回的时候会带上Close header。客户端就知道要close掉connection了。    s.SetKeepAlivesEnabled(false)    s.mu.Lock()    // close listenser    if err := s.l.Close(); err != nil {        return err     }    //将当前idle的connections设置read timeout，便于后续关闭。    t := time.Now().Add(100 * time.Millisecond)    for _, c := range s.conns {        c.SetReadDeadline(t)    }    s.conns = make(map[string]net.Conn)    s.mu.Unlock()    // 等待所有连接结束    s.wg.Wait()    return nil}

Well, through the above method, we can finally shut down the server calmly. But this is only for the connection with the client, the actual MySQL connection, Redis connection, open file handle, and so on, in short, to achieve graceful service close, really is not a very simple thing.