Golang Optimization Road--http Long Connection

This is a creation in Article, where the information may have evolved or changed.

Write in front

There are a lot of connections for the detection of service-side TIME_WAIT status.

netstat -nat | grep :8080 | grep TIME_WAIT | wc -l   17731

TIME_WAITThe state is many, the simple is that the service side actively shut down the TCP connection.

Img-thumbnail

TCP frequently establishes a connection, and there are some problems:

1. Three-time handshake to establish a connection, four handshake disconnection will have a loss of performance;
2. Disconnected disconnection will not be released immediately, will wait 2MSL time, according to my observation is 1 minutes;
3. A large amount of TIME_WAIT memory is used, and a connection measurement is 3.155KB. And take up too much, it is possible to occupy a full port, a server can only have more than 60,000 ports;

TCP-related

The concept of long connections includes TCP long connections and HTTP long connections. The first thing to ensure is that TCP is a long connection. We'll start with that.

func (c *TCPConn) SetKeepAlive(keepalive bool) error

Setkeepalive sets whether the operating system should send keepalive messages on the connection. This method is relatively simple and sets whether to turn on long connections.

func (c *TCPConn) SetReadDeadline(t time.Time) error

Setreaddeadline sets the deadline for future read calls and any currently-blocked read call. A zero value for T means Read would not time out. This function is very fastidious. My previous understanding was to set the read timeout, which is what this means, but there is something else. It is set to the absolute time of the read timeout.

func (c *TCPConn) SetWriteDeadline(t time.Time) error

Setwritedeadline sets the deadline for future write calls and any currently-blocked write call. Even if write times out, it may return n > 0, indicating this some of the data was successfully written. A zero value for T means Write won't time out. This method is to set the write timeout, which is also the absolute time.

How do HTTP packets use TCP long connections?

After the HTTP server starts, it loops through the new requests and creates a process for each request (connection).

// net/http/server.go L1892for {    rw, e := l.Accept()    go c.serve()}

Here's the code for the execution of each of the processes, and I'm just extracting a subset of the key logic. It can be found that there is serve a loop inside the method for .

// net/http/server.go L1320func (c *conn) serve() {    defer func() {        if !c.hijacked() {            c.close()        }    }()    for {        w, err := c.readRequest()        if err != nil {        }        serverHandler{c.server}.ServeHTTP(w, w.req)    }}

What is this cycle for? It is also easy to understand that if it is a long connection, a co-process can perform multiple responses. If it's only done once, it's a short connection. A long connection exits the loop after a time-out or an error, that is, the long connection is closed. deferfunction allows you to close a TCP connection after the end of the process.

readRequestfunction is used to parse the HTTP protocol.

// net/http/server.gofunc (c *conn) readRequest() (w *response, err error) {    if d := c.server.ReadTimeout; d != 0 {        c.rwc.SetReadDeadline(time.Now().Add(d))    }    if d := c.server.WriteTimeout; d != 0 {        defer func() {            c.rwc.SetWriteDeadline(time.Now().Add(d))        }()    }    if req, err = ReadRequest(c.buf.Reader); err != nil {        if c.lr.N == 0 {            return nil, errTooLarge        }        return nil, err    }}func ReadRequest(b *bufio.Reader) (req *Request, err error) {    // First line: GET /index.html HTTP/1.0    var s string    if s, err = tp.ReadLine(); err != nil {        return nil, err    }    req.Method, req.RequestURI, req.Proto, ok = parseRequestLine(s)    mimeHeader, err := tp.ReadMIMEHeader()}

The part that specifically participates in parsing the HTTP protocol is the ReadRequest method, which, before it is called, sets the read and write timeout time. According to the previous description, the time-out is set to absolute time. So it's all time.Now().Add(d) set up here. The difference is that the write timeout is defer executed, that is, after the function returns.

Why does our program have a long connection failure?

Through the source code we can probably know the program flow, according to reason is to support long-connected. Why is our program not working?

Our program uses the Beego framework, which supports timeouts that set both read and write timeouts. And our setting is 1 seconds.

beego.HttpServerTimeOut = 1

My understanding of Read and write timeouts is that the read timeout is the time it takes to get the data to the end of the read, and the write timeout is written from the beginning to the time it was written. My understanding of both timeouts is wrong.

In fact, from the above source can be found that the write timeout is read after the set time-out. That is, the time after the completion of the read, plus the logical execution time, plus the sum of the content return time. In our settings, more than 1 seconds are timed out.

Read the time-out in detail below. ReadRequestis blocked execution, and if no user requests it, it will always wait. The read timeout was ReadRequest set before, and it was a bit time-consuming, in addition to reading the data, which was waiting. If there is no user request, this time the read timeout has been set to 1 seconds later, after more than 1 seconds, the connection will be disconnected.

How to solve the problem?

The reason has been made clear. A large amount TIME_WAIT is caused by a timeout, which may be a read timeout caused by an excessive wait time, or it may be that the program has a partial execution timeout in the case of a voltage test, which results in a write timeout.

We are currently using the Beego framework, which does not support setting the read-write timeout separately, so my current solution is to adjust the read-write timeout a bit larger.

Starting with version 1.6, Golang can support idle timeouts IdleTimeout , which can be considered as the time to read the data, idle timeout to control the wait time. However, it has a problem that if the idle timeout is not set and the read timeout is set, then the read timeout will be used as an idle time-out. I guess the reason for this is to be forward compatible. Another problem is that Beego does not support this time setting, so there is no other way to control the time-out.

Subsequent

In fact, the most reasonable time-out control of the server needs these aspects:

1. Read timeout. is simply a read timeout, do not include the wait time, otherwise you can not distinguish whether the time-out is read data caused by or wait for the cause.
2. Write timeout. It is also best to simply write data timeouts. If the network is good, it is not appropriate to disconnect the connection because the logical execution is slow. The read-write timeout should be set to a shorter setting than the current logical setting.
3. Idle timeout. This can be configured according to the actual situation, can be appropriately larger.

4. Logical timeout. In general, there is no network-level read-write timeout, and the time-out in the case of a voltage test is due to a logical timeout. Golang Native package supported TimeoutHandler . It can control the logic timeout. Unfortunately Beego does not currently support setting a logical timeout. And I didn't think of a good way to get beego into it.

    func TimeoutHandler(h Handler, dt time.Duration, msg string) Handler

Reference documents

• What is the minimum amount of memory per TCP connection in "1" Linux? -Aboutspeaker
• "2" The Complete Guide to Go net/http Timeouts-filippo Valsorda