Several cases of Golang HTTP connection pool failure and analysis of its specific causes

This is a creation in Article, where the information may have evolved or changed.

First, the connection pool fails, the problem is that the background is high-frequency agent,agent will initiate a large number of HTTP requests, but, the net/http is to support the long connection, but, in several cases, have produced a lot of time_wait, here to summarize.

The first case is the misuse of transport, in order to set up the agent, for each request, new has a transport.

client := &http.Client{     CheckRedirect: redirectPolicyFunc,     Timeout: time.Duration(10)*time.Second,//设置超时}client.Transport = &http.Transport{     Proxy: http.ProxyURL(proxyUrl)，} //设置代理ip

The reason for the failure is that the client is thread-safe, the dimension of the Golang connection pool is transport, two maps are maintained in the transport, and the connection is staged.

The second case is not set maxidleconnsperhost, and the connected timeout, once the high-frequency connection exceeds the number of maxidleconnsperhost, while exceeding the timeout, the connection will be released. The correct setting is to instantiate the transport when the connsperhost is evaluated, as follows:

var DefaultTransport RoundTripper = &Transport{        ...   MaxIdleConnsPerHost: 1000,  IdleConnTimeout:       90 * time.Second,        ... }

The third case is that Resp.body forgot to read, directly causing the new request to create the connection directly. In fact, it can be understood that there is no read body socket, if the direct reuse, will produce what kind of consequences? All connections using this socket will be out of character. Example below,

package mainimport (  "fmt"  "html"  "log"  "net"  "net/http"  "time")func startWebserver() {  http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {      fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))  })  go http.ListenAndServe(":8080", nil)}func startLoadTest() {  count := 0  for {      resp, err := http.Get("http://localhost:8080/")      if err != nil {          panic(fmt.Sprintf("Got error: %v", err))      }      resp.Body.Close()      log.Printf("Finished GET request #%v", count)      count += 1  }}func main() {  // start a webserver in a goroutine  startWebserver()  startLoadTest()}

Here you can use Ss-s to view the number of connections, if you do not care about returning body, you can directly discard

 io.Copy(ioutil.Discard, resp.Body)  //Discard 是一个 io.Writer，对它进行的任何 Write 调用都将无条件成功