HTTP long connection 2 million attempts and tuning methods

Transferred from: http://www.linuxde.net/2011/10/1230.html

For a server, we generally consider the QPS that he can support, but there is an application where we need to focus on the number of connections it can support, not the QPS, and of course the QPS is one of the performance points we need to consider. This application is common in message push systems, also known as comet applications, such as chat rooms or instant messaging push systems. The comet application is specifically visible to me in the previous introduction, not much to say here. For such systems, because many messages need to be generated when the client is pushed, so when no message is generated, it is necessary to hold the client's connection, so that when there are a large number of clients, it is necessary to hold a large number of connections, which we call a long connection.

First, we analyze, for this kind of service, need to consume the system resources: CPU, network, memory. So, to achieve the best system performance, we first find the bottleneck of the system. Such a long connection, often we are no data sent, so can also be considered as inactive connections. For the system, this kind of inactive connection does not occupy CPU and network resources, but only consumes the system memory. So, we assume that as long as the system has enough memory, the system will be able to support the number of connections we want to reach, so is the fact true? If this is the case, the kernel will be a test for maintaining this fairly large data structure.

To complete the test, we need to have a server and a large number of clients. Therefore, the service-side program and the client program are required. In order to reach the goal, my idea is this: the client generates a connection, initiates a request to the server, and the server hold the connection without returning the data.

1. Preparation of the service side

For the server side, because of the previous assumptions, we need a large memory servers for the deployment of Nginx comet applications. Here's what I use for the server:

Linux 2.6.18-164.el5 x86_64, 64-bit

Server-side program is simple, based on the Nginx write a comet module, the module accepts the user's request, and then keep the user's connection, not return. Nginx status module, can be directly used to monitor the maximum number of connections.

The service side also needs to adjust the system parameters, in the/etc/sysctl.conf:

Net.core.somaxconn = 2048net.core.rmem_default = 262144net.core.wmem_default = 262144net.core.rmem_max = 16777216net.core.wmem_max = 16777216net.ipv4.tcp_rmem = 4096 4096 16777216net.ipv4.tcp_wmem = 4096 4096 16777216net.ipv4 . Tcp_mem = 786432 2097152 3145728net.ipv4.tcp_max_syn_backlog = 16384net.core.netdev_max_backlog = 20000net.ipv4.tcp_ Fin_timeout = 15net.ipv4.tcp_max_syn_backlog = 16384net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_tw_recycle = 1net.ipv4.tcp_ Max_orphans = 131072

#/sbin/sysctl-p Effective

Here, we mainly look at these items:

NET.IPV4.TCP_RMEM is used to configure the read buffer size, three values, the first is the minimum value of the read buffer, the third is the maximum value, the middle is the default value. We can modify the read buffer size in the program, but not exceed the minimum and maximum. To minimize the amount of memory used by each socket, I set the default value here to 4096.

The NET.IPV4.TCP_WMEM is used to configure the write buffer size.

The read buffer and write buffer in size directly affect the memory footprint of the socket in the kernel.

The NET.IPV4.TCP_MEM is the memory size of TCP, which is the page, not the byte. When the second value is exceeded, TCP enters pressure mode, at which time TCP attempts to stabilize its use of memory, and exits pressure mode when it is less than the first value. When the memory consumption exceeds the third value, TCP will refuse to allocate the socket, view DMESG, will play a lot of logs "tcp:too many of orphaned sockets".

Also net.ipv4.tcp_max_orphans this value to set, this value means that the system can handle the number of sockets that are not part of any process, when we need to quickly establish a large number of connections, we need to pay attention to this value. When the number of sockets that are not part of any process is greater than this value, DMESG will see "Too many of orphaned sockets".

In addition, the server needs to open a large number of file descriptors, such as 2 million, but we set the maximum file descriptor limit, we will encounter some problems, we explain in detail later.

2. Preparation of the client

Since we need to build a large number of clients, we know that on a single system, the local port that is connected to a service is limited. Because the port is a 16-bit integer, it can only be 0 to 65535, and 0 to 1023 is a reserved port, so can allocate only 1024 to 65534, or 64,511. In other words, a machine can only create more than 60,000 long connections. To reach our 2 million connection, we need about 34 clients.

Of course, we can use virtual IP to achieve so many clients, if it is a virtual IP, then each IP can be bound to more than 60,000 ports, 34 virtual IP can be done. And I here, just apply to the company's resources, so the use of physical machines to do.

Due to the system default parameters, the number of automatically assigned ports is from 32768 to 61000, so we need to change the parameters of the client/etc/sysctl.conf:

Net.ipv4.ip_local_port_range = 1024 65535

#/sbin/sysctl-p

The client program is a test program written based on Libevent, which constantly establishes a new connection request.

3. Because the client and the server need to build a large number of sockets, so we need to speed up the maximum file descriptor. Client, need to create more than 60,000 sockets, I set the maximum to 100,000 OK, add in/etc/security/limits.conf:

Admin soft nofile 100000admin hard nofile 100000

Service side, need to create 2 million connection, then I want to set Nofile to 2 million, OK, the problem comes.

When I set nofile to 2 million, the system was unable to log in directly. Try a few times and find that the maximum can only be set to 1 million. After checking the source code, only to know that the original before the 2.6.25 kernel has a macro definition, defined the maximum value of this value, for the 1024*1024, exactly 1 million, and in the 2.6.25 kernel and after that, this value can be set by/proc/sys/fs/nr_open. So I upgraded the kernel to 2.6.32. Ulimit Detailed introduction See Ulimit problem and its influence

After upgrading the kernel, continue with our tuning as follows:

# sudo bash-c 'echo 2000000 >/proc/sys/fs/nr_open '

Now we can set nofile.

Admin soft nofile 2000000admin hard Nofile 2000000

4. Finally, in the process of testing, according to the information of the DMESG system to continuously adjust the configuration in the server/sbin/sysctl, finally, our test completed 2 million long connection. To minimize memory footprint, I changed the Nginx request_pool_size from the default 4k to 1k. In addition, the default values in Net.ipv4.tcp_wmem and Net.ipv4.tcp_rmem are also set to 4k.

2 million when connected, the data is obtained through Nginx monitoring:

2 million system memory condition when connecting:

HTTP long connection 2 million attempts and tuning methods