GNULinux high concurrency performance optimization solution

GNULinux high concurrency performance optimization solution

/**************************************** *******************
* Author: Samson
* Date: 07/14/2015
* Test platform:
* Gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2
* GNU bash, 4.3.11 (1)-release (x86_64-pc-linux-gnu)
* Nginx version:
* Nginx 1.6.2
* Nginx 1.8.0
***************************************** *****************/

GNU Linux high concurrency performance optimization solution

In the GNU Linux system, the main factor affecting the number of connections is determined by the maximum number of files that a single process can open and the number of ports; and The concurrency of a tcp-based server, in addition to the two factors mentioned above, there are also many attributes of the main tcp connection, and the biggest problem is that the connection will remain in the TIME_WAIT status for 60 seconds after the connection is disconnected, this causes that in the case of a large number of high concurrency, there is no available connection when the connection is in the TIME_WAIT status.

1. Modify the port number range:

Default range:

Cat/proc/sys/net/ipv4/ip_local_port_range
32768 61000

As we all know, the port number range is 0 ~ 65535, the well-known port number is between 1 ~ Between 255, 256 ~ The port numbers between 1023 are usually occupied by the system, so if we need more available port numbers, we need to modify the range variable used for the port in the system;

Modification Method: 1) echo "1024 65535">/proc/sys/net/ipv4/ip_local_port_range2), and perform the following settings in/etc/sysctl. conf:

Net. ipv4.ip _ local_port_range = 1024 65535
Run sysctl-p to apply these settings;

3) directly use commands to Optimize System Variables

Sysctl-w net. ipv4.ip _ local_port_range = 1024 65535

If the port is insufficient, an error is returned.

If no idle port is available for use, an error is reported, for example:
Connect () to ip: 80 failed (99: Cannot assign requested address)

Note:

After modifying the port range, if multiple services are deployed on one device, other services start to occupy the "known" ports of other services first, in this case, the service to be listened to is started first, and the "well-known" port used by the Service is occupied first, it won't be difficult.

2. modify the number of files that can be opened by all processes in the system:

Cat/proc/sys/fs/file-max
203466

To modify:
Echo 403466>/proc/sys/fs/file-max

3. For TIME_WAIT processing, you can set the following two items to greatly improve concurrency.

After the communication is complete, the communication connection is recycled almost within seconds. After testing, netstat-ntp will no longer be used to see the connection we just used, but it is described in the official documentation (Default value is 0. it shoshould not be changed without advice/request of technical experts .) you need to be cautious when using these two methods ):
Net. ipv4.tcp _ tw_reuse = 1

// Indicates enabling reuse. Allow TIME-WAIT sockets to be re-used for a New TCP connection. The default value is 0, indicating that the TCP connection is disabled;
Net. ipv4.tcp _ tw_recycle = 1
// Enables fast TIME-WAIT sockets recovery in TCP connections. The default value is 0, indicating that TIME-WAIT sockets is disabled.

Because the above two items are described in the official documents, "It shocould not be changed without advice/request of technical experts. "In other words, these two items may have a negative effect or impact in some cases;

Possible impact:

Net. ipv4.tcp _ tw_recycle is used with net. ipv4.tcp _ timestamps is closely related, while net. ipv4.tcp _ timestamps is enabled by default. When tcp_tw_recycle and tcp_timestamps are both enabled, a hidden property of TCP is activated: The timestamp of the cache connection. Within 60 seconds, if the timestamp of subsequent requests from the same source IP address is smaller than the timestamp in the cache, the kernel will discard the request.

In what scenarios will the timestamp be smaller than the timestamp in the cache?

Similar fault scenarios:
Multiple clients access one server through one NAT, because NAT only changes the IP address information, but does not change the timestamp (the TCP timestamp is not the system time, but the system startup time uptime, therefore, it is unlikely that the TCP Timestamps of the two machines are consistent), so the request will be discarded, so it is easy to cause connection failure.
On the server, tcp_tw_recycle is enabled for rapid fault recovery and Analysis Steps for TIME_WAIT:
1) multiple clients that exit through NAT often request no response from the Web server;
2) Capture packets on the server and find that the server can receive the SYN request from the client, but does not respond to SYN, ACK, that is, the kernel directly discards the packet.

Solution:

1) Disable tcp_timestamps on the service end. The fault can be solved. However, there are security and performance risks. We strongly recommend that you do not disable this variable;
2) Disable tcp_tw_recycle and solve the fault. We recommend that you do not enable this option for machines in the NAT environment;
3) Adjust the network topology to avoid such situations as NAT;
4) The client uses the same NTP service for time synchronization to avoid timestamp differences;

Other optimization parameters

Net. ipv4.tcp _ fin_timeout = 30
// Indicates that if the socket is disabled by the local end, this parameter determines the time it remains in the FIN-WAIT-2 state.
Net. ipv4.tcp _ keepalive_time = 1200
// Indicates the frequency of keepalive messages sent by TCP when keepalive is in use. The default value is 2 hours, which is changed to 20 minutes.
Net. ipv4.tcp _ max_tw_buckets = 5000
// Indicates that the system maintains the maximum number of TIME_WAIT sockets at the same time. If this number is exceeded,
// TIME_WAIT socket is immediately cleared and warning information is printed. The default value is 180000, Which is changed to 5000.
// For servers such as Apache and Nginx, the number of TIME_WAIT sockets can be greatly reduced by parameters in the previous lines,

// This value is the same as/proc/sys/net/ipv4/tcp_max_syn_backlog. It is also a restriction on the backlog parameters in the listen () function. Follow the instructions in this document, it should be best to set the same value as/proc/sys/net/ipv4/tcp_max_syn_backlog. The default value of this value is 128:
Cat/proc/sys/net/core/somaxconn
128
Net. core. somaxconn = 40000

// Specify the maximum length of the unfinished connection queue. The default value is 1024, which is the limit on the number of backlogs in the listen () function of the socket. This value can be increased if the server is overloaded;
Cat/proc/sys/net/ipv4/tcp_max_syn_backlog
1024
Net. ipv4.tcp _ max_syn_backlog = 40000

4. Adjust the maximum file descriptor limit for each process

Adjust the file descriptor limit:

$ Ulimit-n
1024
Modify this value, ulimit-n 4096

$ Vi/etc/security/limits. conf
// Setting Shell Limits for File Descriptors
* Soft nofile 8192
* Hard nofile 8192

The difference between the two is that after the configuration is completed in the/etc/security/limits. conf configuration file, the value of using ulimit-n again after restart is 8192.

5. Re-compile the kernel code to reduce the TIME-wait time for TCP connections

In the include/net/tcp. h file in the kernel code, TIME-WAIT is defined as follows:
// # Define TCP_TIMEWAIT_LEN (60 * HZ)/* how long to wait to destroy TIME-WAIT
* State, about 60 seconds */
You can speed up the release of the connection by modifying the value of TCP_TIMEWAIT_LEN. After the modification, the kernel is compiled and replaced.

The relationship between Nginx configuration and system environment variables is 1024 by default. If worker_connections 4096 is configured in the Nginx configuration file, A warning is displayed:

Nginx: [warn] 4096 worker_connections exceed open file resource limit: 1024
These variables related to the system variables in Nginx are set according to the system configuration. If they are greater than the system variable range, they will not take effect and will be set to the system value by default, for example, the number of files that can be opened by each worker is 1024 by default;

Note:

It is risky to modify the kernel variables. It is best to pass the test in the test environment and then translate the configuration to the production environment.