Golang Development of HTTP services that support smooth upgrade (graceful restart)

Golang Development of HTTP services that support smooth upgrade (graceful restart)

Some time ago using Golang to make an HTTP interface, because of the characteristics of the compiled language, modified the code needs to recompile the executable file, close the running old program, and start a new program. For users with a large number of access to the product, the shutdown, restart the process will inevitably appear inaccessible situation, thereby affecting the user experience.

It is not possible to support a smooth upgrade (graceful restart) when developing an HTTP service using Golang's System package, this article will explore how to solve the problem.

The general idea of smooth upgrade (graceful restart)

In general, to achieve a smooth upgrade, you need the following steps:

1. Replace old executables with new executables (if you just restart gracefully, you can skip this step)

2. Sends a specific signal to the running old process via PID (KILL-SIGUSR2 $pid)

3. A running old process that receives a specified signal, starts a new executable file as a child process, and starts processing a new request

4. The old process no longer accepts new requests, waits for unfinished service to finish processing, and then ends normally

5. The new process is adopted by the INIT process after the parent process exits and continues to serve

Second, Golang Socket network programming

Sockets are the package and application of TCP/IP for the Transport Layer protocol at the programmer level. Golang socket-related functions and structures in the net package, we learn from a simple example of Golang socket network programming, the key instructions are written directly in the comments.

1. Service-side program Server.go

package mainimport ("fmt""log""net""time")func main() {// 监听8086端口listener, err := net.Listen("tcp", ":8086")if err != nil {log.Fatal(err)}defer listener.Close()for {// 循环接收客户端的连接，没有连接时会阻塞，出错则跳出循环conn, err := listener.Accept()if err != nil {fmt.Println(err)break}fmt.Println("[server] accept new connection.")// 启动一个goroutine 处理连接go handler(conn)}}func handler(conn net.Conn) {defer conn.Close()for {// 循环从连接中 读取请求内容，没有请求时会阻塞，出错则跳出循环request := make([]byte, 128)readLength, err := conn.Read(request)if err != nil {fmt.Println(err)break}if readLength == 0 {fmt.Println(err)break}// 控制台输出读取到的请求内容，并在请求内容前加上hello和时间后向客户端输出fmt.Println("[server] request from ", string(request))conn.Write([]byte("hello " + string(request) + ", time: " + time.Now().Format("2006-01-02 15:04:05")))}}

2. Client program Client.go

package mainimport ("fmt""log""net""os""time")func main() {// 从命令行中读取第二个参数作为名字，如果不存在第二个参数则报错退出if len(os.Args) != 2 {fmt.Fprintf(os.Stderr, "Usage: %s name ", os.Args[0])os.Exit(1)}name := os.Args[1]// 连接到服务端的8086端口conn, err := net.Dial("tcp", "127.0.0.1:8086")checkError(err)for {// 循环往连接中 写入名字_, err = conn.Write([]byte(name))checkError(err)// 循环从连接中 读取响应内容，没有响应时会阻塞response := make([]byte, 256)readLength, err := conn.Read(response)checkError(err)// 将读取响应内容输出到控制台，并sleep一秒if readLength > 0 {fmt.Println("[client] server response:", string(response))time.Sleep(1 * time.Second)}}}func checkError(err error) {if err != nil {log.Fatal("fatal error: " + err.Error())}}

3. Run the sample program

# 运行服务端程序go run server.go# 在另一个命令行窗口运行客户端程序go run client.go "tabalt"

Third, Golang HTTP programming

HTTP is an application-layer protocol based on the Transport Layer protocol TCP/IP. HTTP-related implementations in the Golang are used in the Net/http package to directly use the socket-related functions and structures in the net package.

Let's learn from a simple example of Golang HTTP programming, with key instructions written directly in the comments.

1. HTTP Service Program Http.go

package mainimport ("log""net/http""os")// 定义http请求的处理方法func handlerHello(w http.ResponseWriter, r *http.Request) {w.Write([]byte("http hello on golang\n"))}func main() {// 注册http请求的处理方法http.HandleFunc("/hello", handlerHello)// 在8086端口启动http服务，会一直阻塞执行err := http.ListenAndServe("localhost:8086", nil)if err != nil {log.Println(err)}// http服务因故停止后 才会输出如下内容log.Println("Server on 8086 stopped")os.Exit(0)}

2. Run the sample program

# 运行HTTP服务程序go run http.go# 在另一个命令行窗口curl请求测试页面curl http://localhost:8086/hello/# 输出如下内容：http hello on golang

IV. implementation of socket operation in Golang net/http package

From the simple example above, we see that in Golang to start an HTTP service, we need only three simple steps:

1. Defining how HTTP requests are handled

2. How to register HTTP request processing

3. To start the HTTP service on a port

And the most critical start of the HTTP service is to invoke HTTP. Implemented by the Listenandserve () function. Here we find the implementation of the function:

func ListenAndServe(addr string, handler Handler) error {server := &Server{Addr: addr, Handler: handler}return server.ListenAndServe()}

Here we create a server object and call its Listenandserve () method, and we find the implementation of the Listenandserve () method of the struct server:

func (srv *Server) ListenAndServe() error {addr := srv.Addrif addr == "" {addr = ":http"}ln, err := net.Listen("tcp", addr)if err != nil {return err}return srv.Serve(tcpKeepAliveListener{ln.(*net.TCPListener)})}

As you can see from the code, the TCP port is listening and the listener is packaged into a struct tcpkeepalivelistener and then SRV is called. Serve () method; We continue to follow the implementation of the Serve () method:

func (srv *Server) Serve(l net.Listener) error {defer l.Close()var tempDelay time.Duration // how long to sleep on accept failurefor {rw, e := l.Accept()if e != nil {if ne, ok := e.(net.Error); ok && ne.Temporary() {if tempDelay == 0 {tempDelay = 5 * time.Millisecond} else {tempDelay *= 2}if max := 1 * time.Second; tempDelay > max {tempDelay = max}srv.logf("http: Accept error: %v; retrying in %v", e, tempDelay)time.Sleep(tempDelay)continue}return e}tempDelay = 0c, err := srv.newConn(rw)if err != nil {continue}c.setState(c.rwc, StateNew) // before Serve can returngo c.serve()}}

As you can see, like the example code in front of our socket programming, loops are connected from the listening port, if a net is returned. Error and the bug is temporary, it will sleep for a time before continuing. If another error is returned, the loop is terminated. After the successful accept to a connection, called the Method Srv.newconn () to make a layer of the connection wrapper, and finally a goroutine processing HTTP request.

V. Golang smooth upgrade (graceful restart) Implementation of HTTP service

I created a new package gracehttp to implement HTTP services that support a smooth upgrade (graceful restart), in order to write less code and reduce usage costs, the new package uses as much of the net/http package implementation as possible and net/http maintains a consistent external approach to the package. Now let's look gracehttp at the package support for smooth upgrade (graceful restart) Golang HTTP service involves the details of how to implement.

1, Golang processing signal

The Golang os/signal package encapsulates the processing of the signal. For simple usage, see the example:

package mainimport ("fmt""os""os/signal""syscall")func main() {signalChan := make(chan os.Signal)// 监听指定信号signal.Notify(signalChan,syscall.SIGHUP,syscall.SIGUSR2,)// 输出当前进程的pidfmt.Println("pid is: ", os.Getpid())// 处理信号for {sig := <-signalChanfmt.Println("get signal: ", sig)}}

2, the child process to start a new program, listening to the same port

In the implementation code of the fourth part of the Listenandserve () method, you can see that the Net/http package uses net.Listen functions to listen on a port, but if a running program is already listening on a port, other programs cannot listen to the port. The solution is to start with a child process and pass the file descriptor of the listener port to the child process, which implements the port listener from the file descriptor.

The implementation needs to use an environment variable to distinguish whether the process is starting normally or as a child process, the relevant code excerpt is as follows:

Start child process Execute new program func (This *server) startnewprocess () error {LISTENERFD, err: = This.listener. ( *listener). GETFD () if err! = Nil {return FMT. Errorf ("Failed to get socket file descriptor:%v", err)}path: = OS. args[0]//sets an environment variable that identifies graceful restarts environlist: = []string{}for _, Value: = Range OS. Environ () {if value! = graceful_environ_string {environlist = append (environlist, value)}}environlist = Append ( Environlist, graceful_environ_string) Execspec: = &syscall. Procattr{env:environlist,files: []uintptr{os. STDIN.FD (), OS. STDOUT.FD (), OS. STDERR.FD (), listenerfd},}fork, err: = Syscall. Forkexec (Path, OS. Args, Execspec) if err! = Nil {return FMT. Errorf ("Failed to Forkexec:%v", err)}this.logf ("Start new process success, PID%d.", fork) return Nil}func (this *server) Getnettcplistener (addr string) (*net. TcpListener, error) {var ln net. Listenervar Err Errorif this.isgraceful {file: = os. NewFile (3, "") ln, err = net. Filelistener (file) if err! = Nil {err = FMT. Errorf ("net. Filelistener error:%v ", err) return nil, ERR}} else {ln, err = net. Listen ("tcp", addr) if err! = Nil {err = FMT. Errorf ("net. Listen error:%v ", err) return nil, Err}}return Ln. (*net. TcpListener), nil}

3. The parent process waits for an outstanding request in an existing connection to finish processing

This piece is the most complex; first we need a counter, when a connection is successfully accept, the counter adds 1, the count minus 1 when the connection is closed, the counter is 0, the parent process can exit normally. The Waitgroup in the Golang's sync bag is a good way to achieve this.

Then to control the establishment and shutdown of the connection, we need to drill down into the serve () method of the server structure in the Net/http package. Revisiting the implementation of the fourth serve () method, you will find it almost impossible to rewrite a serve () method, because this method calls a lot of non-exportable internal methods, overriding the serve () method almost rewrites the entire net/http package.

Fortunately, we also found that we passed a listener to the serve () method in the Listenandserve () method, and finally called the Listener accept () method, which returned a conn example. Finally, the close () method of Conn is called when the connection is broken, and these structures and methods are exportable!

We can define our own listener structure and conn structure, combine the net/http corresponding structure in the package, and rewrite the Accept () and close () methods to achieve a count of the connections, the relevant code excerpt as follows:

type Listener struct {*net.TCPListenerwaitGroup *sync.WaitGroup}func (this *Listener) Accept() (net.Conn, error) {tc, err := this.AcceptTCP()if err != nil {return nil, err}tc.SetKeepAlive(true)tc.SetKeepAlivePeriod(3 * time.Minute)this.waitGroup.Add(1)conn := &Connection{Conn:     tc,listener: this,}return conn, nil}func (this *Listener) Wait() {this.waitGroup.Wait()}type Connection struct {net.Connlistener *Listenerclosed bool}func (this *Connection) Close() error {if !this.closed {this.closed = truethis.listener.waitGroup.Done()}return this.Conn.Close()}

4, the use of Gracehttp package

The Gracehttp package has been applied to hundreds of millions of PV projects per day and is open source to GitHub: Github.com/tabalt/gracehttp is very simple to use.

As the following sample code, after the introduction of the package just modify one keyword, the http. Listenandserve changed to Gracehttp. Listenandserve can be.

package mainimport (    "fmt"    "net/http"    "github.com/tabalt/gracehttp")func main() {    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {        fmt.Fprintf(w, "hello world")    })    err := gracehttp.ListenAndServe(":8080", nil)    if err != nil {        fmt.Println(err)    }}

To test the effect of a smooth upgrade (graceful restart), refer to the following page for instructions:

Https://github.com/tabalt/gracehttp#demo

Golang Development of HTTP services that support smooth upgrade (graceful restart)