GNU Linux high Concurrency performance optimization scheme

/***********************************************************
* Author:samson
* date:07/14/2015
* Test Platform:
* GCC (Ubuntu 4.8.2-19ubuntu1) 4.8.2
* GNU Bash, 4.3.11 (1)-release (X86_64-PC-LINUX-GNU)
* Nginx Version:
* Nginx 1.6.2
* Nginx 1.8.0
* *********************************************************/

GNU Linux high Concurrency performance optimization scheme

In the GNU Linux system, the factors that affect the number of connections are determined by the maximum number of files, the number of ports that a single process can open, and the concurrency of a TCP-based server, in addition to the two factors mentioned above, as well as the many attributes of the primary TCP connection. The biggest problem is that the connection disconnects after the TIME_WAIT state has been in existence for 60 seconds, resulting in a large number of high concurrency when the connection to this time_wait state is not available.

1, modify the port number range:

Default range:

Cat/proc/sys/net/ipv4/ip_local_port_range
32768 61000

As we all know, the port number range is 0~65535, the well-known port number is between 1~255, the port number between 256~1023 is usually occupied by the system, so we need more to use the port number, then we need to modify the system to use the range of ports variable;

Modify the method: 1), echo "65535" >/proc/sys/net/ipv4/ip_local_port_range2), in/etc/sysctl.conf the following settings:

net.ipv4.ip_local_port_range=1024 65535
Then execute: sysctl-p These settings to take effect;

3), direct use of the command to optimize the system variables

Sysctl-w net.ipv4.ip_local_port_range=1024 65535

If the port is not enough to use the error message

If there is no free port to use, it will be an error, such as:
Connect () to ip:80 failed (99:cannot assign requested address)

Attention:

After modifying the scope of the port, if there are multiple services on one device, if the other service first started by other services "well-known" Port to occupy, then this problem is less good processing, in this case, the need to listen to the service to start, the service to use the "well-known" Port, there will be no more hassle.

2. Modify the number of files that can be opened by all processes in the system:

Cat/proc/sys/fs/file-max
203466

To modify the words:
echo 403466 >/proc/sys/fs/file-max

3, for the problem of handling time_wait, by setting the following two items can greatly improve the concurrency

After the communication has been completed, the connection of the completion of the communication has been collected in the second level, tested, and then using NETSTAT-NTP, will not see the connection just used, but in the official documentation (Default value is 0. It should not be changed without advice/request of technical experts.) Use these two methods to be very cautious, as follows (see how to modify the method above):
Net.ipv4.tcp_tw_reuse = 1

means to turn on reuse. Allows time-wait sockets to be re-used for new TCP connections, which defaults to 0, which means shutdown;
Net.ipv4.tcp_tw_recycle = 1
Represents a quick recycle of time-wait sockets on a TCP connection, which defaults to 0, which means shutdown.

Since the above two items are described in the official documentation "It should not being changed without advice/request of technical experts.", that is to say, these two will have negative effects or effects in some cases;

Possible effects:

Net.ipv4.tcp_tw_recycle is closely related to Net.ipv4.tcp_timestamps, while Net.ipv4.tcp_timestamps is turned on by default, when Tcp_tw_recycle and Tcp_ Timestamps a hidden property that activates TCP when it is opened simultaneously: the timestamp of the cached connection. Within 60 seconds, the timestamp of subsequent requests from the same source IP is less than the timestamp in the cache, and the kernel discards the request.

What scenario would make the timestamp less than the timestamp in the cache?

A similar failure scenario:
Multiple clients access a server through a NAT, because NAT only changes the IP address information, but does not change the timestamp (TCP timestamp is not the system time, but the system boot time uptime, so the TCP timestamp of the two machines is less likely to be consistent), Then there will be a situation where the request is discarded, so it is easy to create a connection failure.
The fast recovery symptom and analysis steps for time_wait are turned on on the server: tcp_tw_recycle
1) Multiple clients through the NAT exit often request that the Web server be unresponsive;
2) in the server grab packet, found that the service side can receive the client's SYN request, but did not respond to Syn,ack, that is, the kernel directly dropped the packet.

Workaround:

1) Close the service to its end of the tcp_timestamps, the fault can be resolved, but there are security and performance risks, it is strongly recommended not to close this variable;
2) Close the tcp_tw_recycle, the fault can also be resolved. It is recommended that the machine in the NAT environment do not turn on this option;
3) Adjust the network topology to avoid this similar situation of NAT;
4) The client uses the same NTP service for time synchronization to avoid timestamp differences in time synchronization;

Other optimization parameters

Net.ipv4.tcp_fin_timeout = 30
Indicates that if the socket is closed by a local requirement, this parameter determines when it remains in the fin-wait-2 state.
Net.ipv4.tcp_keepalive_time = 1200
Indicates the frequency at which TCP sends keepalive messages when KeepAlive is employed. The default is 2 hours, which is changed to 20 minutes.
Net.ipv4.tcp_max_tw_buckets = 5000
Indicates that the system maintains the maximum number of time_wait sockets at the same time, if this number is exceeded,
The time_wait socket is immediately cleared and a warning message is printed. The default is 180000, which changes to 5000.
For Apache, Nginx and other servers, the parameters of the last few lines can be a good way to reduce the number of time_wait sockets,

This value is the same as/proc/sys/net/ipv4/tcp_max_syn_backlog, and is also a limitation on the backlog parameter in the Listen () function, which, as described in the documentation, should preferably be set to and/proc/sys/net/ Ipv4/tcp_max_syn_backlog the same value, the default value for this value is 128:
Cat/proc/sys/net/core/somaxconn
128
Net.core.somaxconn = 40000

Specifies the maximum length of the outstanding connection queue, which defaults to 1024, which is the limit on the number of backlogs in the socket's listen () function, which can be increased if the server is overloaded;
Cat/proc/sys/net/ipv4/tcp_max_syn_backlog
1024
Net.ipv4.tcp_max_syn_backlog = 40000

4. Adjust maximum open file descriptor limit per process

To adjust file descriptor limits:

$ ulimit-n
1024
Modify this value, ulimit-n 4096

$vi/etc/security/limits.conf
Setting Shell Limits for File descriptors
*soft Nofile 8192
*hard Nofile 8192

The difference between the two is that after being configured in the/etc/security/limits.conf configuration file, using Ulimit-n again after a reboot will get a value of 8192.

5. Reduce the time-wait time for TCP connections by recompiling kernel code

In the kernel code of the Include/net/tcp.h file, time-wait is defined as follows:
#define Tcp_timewait_len (60*hz)/* How long to wait to destroy time-wait
* State, about seconds */
You can speed up the release of a connection by modifying the value of the Tcp_timewait_len, and then after the kernel is compiled and replaced.

Nginx configuration and system environment variables of the relationship between the system default is 1024, if in Nginx configuration file, configured worker_connections 4096, and then start, there will be a warning:

Nginx: [warn] 4096 worker_connections exceed Open file resource limit:1024
In Nginx, these and system variables are set according to the configuration of the system, if it is larger than the scope of the system variable, will not take effect, the default is the value of the system, such as each worker can open the number of files will be default to the system value 1024;

Attention:

It is risky to modify kernel variables, preferably in a test environment, and then translate the configuration into a production environment.

REF:

The meaning and value of the kernel parameters of IPV4 and IPV6:
Https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
An introduction to the main items under the/proc directory:
Http://man7.org/linux/man-pages/man5/proc.5.html

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

GNU Linux high Concurrency performance optimization scheme