Long connection short connection long

Long connection short connection long

This is still an old problem:

Record one file download packet loss fill-in Trip http://www.cnblogs.com/syjkfind/p/5281677.html

Even if only the haproxy-nginx-disk files are relatively small, the files are very large and sometimes incomplete.

From the perspective of symptom, the browser response is 200 OK, and the log of the curl command is displayed as curl: (18) transfer closed with 204800 bytes remaining to read

Literally, the connection is closed. There is no relevant answer to query relevant information. It is ruled out that the nginx buffer size is different, but the connection is occasionally closed.

I was puzzled until I went to The haproxy configuration several times with my O & M colleagues... A keyword caught my attention: http-server-close

[Http-server-close]

Configuration reference http://www.cnblogs.com/dkblog/archive/2012/03/13/2393321.html

The general idea is that haproxy maintains a persistent connection with the browser, but haproxy uses a short connection with the backend server.

What is persistent connection reference http://www.cnblogs.com/cswuyg/p/3653263.html?

So is it a persistent connection or a short connection? It depends on specific scenarios. Persistent connections can save the overhead of repeated connections and speed up response. Transient connections can speed up connection release and improve concurrency.

So I changed it to http-pretend-keepalive, which seems to be a lot fewer, and later gzip transmission will rarely happen again. Of course, the occasional occurrence of the problem is also a problem of network speed. If there is no solution, you cannot increase the timeout without limit.

In addition, haproxy mentioned above should use Layer 7 http or Layer 3 tcp. Most of the information is about http. Here we mention that tcp mode is used for https forwarding. Http://serverfault.com/questions/611272/haproxy-http-vs-tcp

[Resumable upload]

After the change to http-pretend-keepalive, interesting things happen, and the browser returns 206. That is to say, in the case of persistent connections, resumable data transfer is supported ~~~

[Enable gzip Transmission]

Nginx is enabled to support gzip transmission, but it does not take effect. Why? The reason is that the response type is incorrect. The supported gzip formats are plain text and html, but the default type is application/octet-stream. You only need to set the type of the returned file to text/plain. The configuration of enabling gzip and setting Content-Type is not described in detail.

Of course, it is amazing to enable gzip. The original 20 MB text file is only 1.5 MB! Of course, there is also a small problem, that is, the transfer encoding is chunked, and you cannot know the total size of the file, and whether the file is complete. However, when the bandwidth is reduced by 92%, the transmission fails even if the user has not heard of it.

The extracted file is the correct text with less content.

 

Reflecting on the whole process, we may feel a bit of luck. If we have a good packet capture or monitoring experience on the haproxy and nginx layers, the problem can be located more quickly and accurately (but in the production environment, you know ). What's more, the network link is a ghost, and how to reproduce it is also a challenge. Fortunately, I found a way out in "bold assumptions and careful proof" by referring to various materials and configuring various "Iot maps" in logs.

So far, the problem has been solved relatively satisfactorily.