HTTP long connection 2 million attempts and tuning

For a server, we generally consider the QPS that he can support, but there is an application where we need to focus on the number of connections it can support, not the QPS, and of course the QPS is one of the performance points we need to consider. This application is common in message push systems, also known as comet applications, such as chat rooms or instant messaging push systems. The comet application is specifically visible to me in the previous introduction, not much to say here. For such systems, because many messages need to be generated when the client is pushed, so when no message is generated, it is necessary to hold the client's connection, so that when there are a large number of clients, it is necessary to hold a large number of connections, which we call a long connection.   First of all, we analyze the system resources that are consumed by this type of service: CPU, network, memory. So, to achieve the best system performance, we first find the bottleneck of the system. Such a long connection, often we are no data sent, so can also be considered as inactive connections. For the system, this kind of inactive connection does not occupy CPU and network resources, but only consumes the system memory. So, we assume that as long as the system has enough memory, the system will be able to support the number of connections we want to reach, so is the fact true? If this is the case, the kernel will be a test for maintaining this fairly large data structure.   To complete the test, we need to have a server and a large number of clients. Therefore, the service-side program and the client program are required. In order to reach the goal, my idea is this: the client generates a connection, initiates a request to the server, and the server hold the connection without returning the data.  1. Server-side readiness for the service side, because of the previous assumptions, we need a large memory servers for deploying Nginx comet applications. Here's what I use for the server: Summary:        dell R710, 2 x Xeon E5520 2.27GHz, 23.5GB/24GB 1333MHz  system: &N Bsp       Dell PowerEdge R710 (Dell 0VWN1R)  processors:     2 x Xeon E5520 2.27GHz 5860MHz FSB (cores)  memory:         23.5GB/24GB 1333MHz = = 6 x 4GB, x empty  disk-control: &nbs P Megaraid_sas0:dell/lsilogic PERC 6/i, Package 6.2.0-0013, FW 1.22.02-0612,  network:        eth 0 (BNX2): Broadcom netxtreme II BCM5709 Gigabit ethernet,1000mb/s  os:             RHEL Server 5.4 (Tikanga), Linux 2.6.18-164.el5 x86_64, 64-bit   Server program is very simple, based on the Nginx write a comet module, the module accepts the user's request, and then keep the user's connection, not return. Nginx status module, can be directly used to monitor the maximum number of connections.   Server also need to adjust the system parameters, in/etc/sysctl.conf: Net.core.somaxconn = 2048  net.core.rmem_default = 262144   Net.core.wmem_default = 262144  net.core.rmem_max = 16777216  net.core.wmem_max = 16777216  net.ipv4.tcp _rmem = 4096 4096 16777216  net.ipv4.tcp_wmem = 4096 4096 16777216  net.ipv4.tcp_mem = 786432 2097152 3145728 &n Bsp;net.ipv4.tcp_max_syn_backlog = 16384  net.core.netdev_max_backlog = 20000  net.ipv4.tcp_fin_timeout = 15  net.ipv4.tcp_max_syn_backlog = 16384  net.ipv4.tcp_tw_reuse = 1  net.ipv4.tcp_tw_recycle = 1   Net.ipv4.tcp_max_orphans= 131072  /sbin/sysctl-p Effective   Here, we mainly look at these items: Net.ipv4.tcp_rmem used to configure the size of the read buffer, three values, the first is the minimum value of the read buffer, the third is the maximum value, the middle is the default value. We can modify the read buffer size in the program, but not exceed the minimum and maximum. To minimize the amount of memory used by each socket, I set the default value here to 4096. The NET.IPV4.TCP_WMEM is used to configure the write buffer size. The read buffer and write buffer in size directly affect the memory footprint of the socket in the kernel. The NET.IPV4.TCP_MEM is the memory size of TCP, which is the page, not the byte. When the second value is exceeded, TCP enters pressure mode, at which time TCP attempts to stabilize its use of memory, and exits pressure mode when it is less than the first value. When the memory consumption exceeds the third value, TCP will refuse to allocate the socket, view DMESG, will play a lot of logs "tcp:too many of orphaned sockets". Also net.ipv4.tcp_max_orphans this value to set, this value means that the system can handle the number of sockets that are not part of any process, when we need to quickly establish a large number of connections, we need to pay attention to this value. When the number of sockets that are not part of any process is greater than this value, DMESG will see "Too many of orphaned sockets". In addition, the server needs to open a large number of file descriptors, such as 2 million, but we set the maximum file descriptor limit, we will encounter some problems, we explain in detail later.  2. Client preparation because we need to build a large number of clients, we know that on a single system, the local port that is connected to a service is limited. Because the port is a 16-bit integer, it can only be 0 to 65535, and 0 to 1023 is a reserved port, so can allocate only 1024 to 65534, or 64,511. In other words, a machine can only create more than 60,000 long connections. To reach our 2 million connection, we need about 34 clients.   Of course, we can use virtual IP to achieve so many clients, if it is a virtual IP, then each IP can be bound to more than 60,000 ports, 34 virtual IP can be done. And I here, just apply to the company's resources, so the use of physical machines to do.   Due to the system default parameters, the number of automatically assigned ports is from 32768 to 61000, so we need to change the parameters of the client/etc/sysctl.conf: net.ipv4.ip_lOcal_port_range = 65535  /sbin/sysctl-p  client program is a test program based on Libevent, which constantly establishes new connection requests.  3. Because the client and the server need to build a large number of sockets, so we need to speed up the maximum file descriptor. Client, need to create more than 60,000 sockets, I set the maximum to 100,000 OK, add in/etc/security/limits.conf: admin    soft    nofile   100000  admin    hard    nofile  100000    service side, need to create 2 million connections, then I want to set Nofile to 2 million, OK, Here's the problem. When I set nofile to 2 million, the system was unable to log in directly. Try a few times and find that the maximum can only be set to 1 million. After checking the source code, only to know that the original before the 2.6.25 kernel has a macro definition, defined the maximum value of this value, for the 1024*1024, exactly 1 million, and in the 2.6.25 kernel and after that, this value can be set by/proc/sys/fs/nr_open. So I upgraded the kernel to 2.6.32.   After upgrading the kernel, continue with our tuning as follows: sudo bash-c ' echo 2000000 >/proc/sys/fs/nr_open '    now set nofile on: admin   & Nbsp;soft    nofile  2000000  admin    hard    nofile  2000000   Finally, in the process of testing, according to the information of the DMESG system to continuously adjust the configuration in the service-side/sbin/sysctl, finally our test completed 2 million long connection. To minimize memory footprint, I changed the Nginx request_pool_size from the default 4k to 1k. In addition, the default values in Net.ipv4.tcp_wmem and Net.ipv4.tcp_rmem are also set to 4k. 2 million when connected, the data is obtained through Nginx monitoring:

2 million system memory condition when connecting:

HTTP long connection 2 million attempts and tuning