GNU Linux high Concurrency performance optimization scheme

/***********************************************************
* Author:samson
* date:07/14/2015
* Test Platform:
* GCC (Ubuntu 4.8.2-19ubuntu1) 4.8.2
* GNU Bash, 4.3.11 (1)-release (X86_64-PC-LINUX-GNU)
* Nginx Version:
* Nginx 1.6.2
* Nginx 1.8.0
* *********************************************************/

GNU Linux high Concurrency performance optimization scheme

In the GNU Linux system, the factors that affect the number of connections are determined by the maximum number of files, the number of ports that a single process can open, and the concurrency of a TCP-based server, in addition to the two factors mentioned above, as well as a very large number of properties for basic TCP connections. The biggest problem is that the connection disconnects after the TIME_WAIT state has been in existence for 60 seconds, resulting in a large number of high concurrency when the connection to this time_wait state is not available.

1. Change the port number range:

Default range:

Cat/proc/sys/net/ipv4/ip_local_port_range
32768 61000

As we all know, the port number range is 0~65535, the well-known port number is between 1~255. The port number between 256~1023 is usually occupied by the system, so we need a lot of other ports that can be used, so we need to change the use of the port range variable in the system;

Change method: 1), echo "65535" >/proc/sys/net/ipv4/ip_local_port_range2), in/etc/sysctl.conf for example, the following settings:

net.ipv4.ip_local_port_range=1024 65535
Then run: sysctl-p These settings to take effect;

3), direct use of the command to optimize the system variables

Sysctl-w net.ipv4.ip_local_port_range=1024 65535

Error message if port is not available

If there is no spare port to use. will be an error. Such as:
Connect () to ip:80 failed (99:cannot assign requested address)

Attention:

After changing the range of the port. If there are multiple services on a single device. If first started by other services will be another service "well-known" Port to occupy, then this problem is relatively less merit rationale, in this case, for the need to listen to the service to start first, the service to use the "well-known" Port first occupation. There will be no more trouble than the situation.

2. Change the number of files that can be opened by all processes in the system:

Cat/proc/sys/fs/file-max
203466

To change the words:
echo 403466 >/proc/sys/fs/file-max

3, for the problem of handling time_wait, by setting the following two items can greatly improve the concurrency

After the communication has been completed, the communication is almost the same connection in the second level between the recovery, tested, and then use NETSTAT-NTP, will not see the connection just used, but in the official documents explained (Default value is 0. It should not being changed without advice/request of technical experts. Use these two methods to be very cautious, such as the following (change method please refer to the above change way):
Net.ipv4.tcp_tw_reuse = 1

means to turn on reuse. Agree to the Time-wait sockets again for the new TCP connection, default 0, indicating that the shutdown;
Net.ipv4.tcp_tw_recycle = 1
Indicates the high-speed recovery of the time-wait sockets in the TCP connection is turned on, and defaults to 0, indicating shutdown.

Since these two items are described in official documents, "it should not being changed without advice/request of technical experts.", that is to say, these two items will have negative effects or effects in some cases.

Possible effects:

Net.ipv4.tcp_tw_recycle is closely related to Net.ipv4.tcp_timestamps, while Net.ipv4.tcp_timestamps is turned on by default, when Tcp_tw_recycle and Tcp_ Timestamps a hidden property that activates TCP when it is opened at the same time: the timestamp of the cached connection.

Within 60 seconds, the requested timestamp of the same source IP is less than the timestamp in the cache, and the kernel discards the request.

What scenario would make the timestamp less than the timestamp in the cache?

Similar failure scenarios:
Multiple clients access a single server through a NAT, because NAT only changes the IP address information, but does not change the timestamp (TCP timestamp is not the system time.) It is the time uptime the system starts. So the probability of a TCP timestamp of two machines is very small). Then there will be a situation where the request is discarded, so very easy causes the connection to fail.

The tcp_tw_recycle for time_wait high-speed recovery failure and analysis steps are enabled on the server:
1) Multiple clients through the NAT exit often request that the webserver be unresponsive;
2) The server catches the packet and discovers that the service side can receive the client's SYN request. However, there is no response to Syn,ack, that is, the kernel drops the package directly.

Workaround:

1) Close the service to its end of the tcp_timestamps. Failures can be solved, but there are security and performance implications. It is strongly recommended not to close this variable;
2) Close the tcp_tw_recycle, the fault can also be solved. It is recommended that the machine in the NAT environment do not turn on this option;
3) Adjust the network topology to avoid this similar situation for NAT.
4) The client uses the same NTP service for time synchronization. Make time synchronization avoid timestamp difference;

Other optimization parameters

Net.ipv4.tcp_fin_timeout = 30
Indicates that the socket is closed by this side requirement. This parameter determines how long it remains in the fin-wait-2 state.
Net.ipv4.tcp_keepalive_time = 1200
Indicates the frequency at which TCP sends keepalive messages when KeepAlive is employed. The default is 2 hours, which is changed to 20 minutes.
Net.ipv4.tcp_max_tw_buckets = 5000
Indicates that the system maintains the maximum number of time_wait sockets at the same time, assuming that this number is exceeded,
The time_wait socket is immediately cleared and the warning message is printed. Default 180000, change to 5000.

For Apache, Nginx and other servers, the last few lines of the number of parameters can be very good to reduce the number of time_wait sockets.

This value is the same as/proc/sys/net/ipv4/tcp_max_syn_backlog, and is also a limitation of the backlog in the listen () function, which, according to the documentation, should preferably be set to and/proc/sys/net/ Ipv4/tcp_max_syn_backlog the same value, the default value for this value is 128:
Cat/proc/sys/net/core/somaxconn
128
Net.core.somaxconn = 40000

Specifies the maximum length of the outstanding connection queue, which defaults to 1024, which is the limit on the backlog number in the socket's listen () function, which can be increased if the server overload is possible;
Cat/proc/sys/net/ipv4/tcp_max_syn_backlog
1024
Net.ipv4.tcp_max_syn_backlog = 40000

4. Adjust the maximum open File Description descriptor limit for each process

Adjust file Description Descriptor Restrictions:

$ ulimit-n
1024
Change this value. Ulimit-n 4096

$vi/etc/security/limits.conf
Setting Shell Limits for File descriptors
*soft Nofile 8192
*hard Nofile 8192

The difference between the two, after being configured in the/etc/security/limits.conf configuration file, the value of using Ulimit-n again after restarting is 8192.

5. Reduce time-wait time for TCP connections by compiling kernel code again

In the Include/net/tcp.h file in the kernel code. The definition of time-wait is as follows:
#define Tcp_timewait_len (60*hz)/* How long to wait to destroy time-wait
* State, about seconds */
Ability to change the value of the Tcp_timewait_len. To expedite the release of the connection; After the change, the kernel is compiled and replaced.

The relationship between Nginx configuration and system environment variables system default is 1024. If worker_connections 4096 is configured in the Nginx configuration file, a warning will appear after the boot:

Nginx: [warn] 4096 worker_connections exceed Open file resource limit:1024
These in nginx are related to system variables and are set according to the configuration in the system. If it is larger than the range of the system variable, it will not take effect, default to the system value, such as the number of files that each worker can open is defaulted to the system value 1024;

Attention:

It is risky to change the kernel variables, preferably in a test environment. Then pan the configuration to the production environment.

REF:

The meaning and value of the kernel parameters of IPV4 and IPV6:
Https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
An introduction to the main items under the/proc folder:
Http://man7.org/linux/man-pages/man5/proc.5.html

GNU Linux high Concurrency performance optimization scheme