How do you optimize the number of connections for a Linux server with a large request?

The default Linux server file descriptor, and so on, opens up to 1024, with ulimit-a view:

[Viewuser@~]$ ulimit-a
Core file size (blocks,-c) 0 #coredump files Size
Data seg Size (Kbytes,-D) Unlimited
Scheduling Priority (-e) 0
File size (blocks,-f) Unlimited
Pending Signals (I.) 255622
Max locked Memory (Kbytes, L) 64
Max memory Size (Kbytes, M) Unlimited
Open files (-N) 1024 #打开文件数量, Root account Unlimited
Pipe Size (bytes, p) 8
POSIX message queues (bytes,-Q) 819200
Real-time priority (-R) 0
Stack size (Kbytes,-s) 8192
CPU time (seconds,-t) unlimited
MAX User Processes (-u) 4096 #root用户本项是无限
Virtual Memory (Kbytes,-V) Unlimited
File locks (-X) Unlimited

If you exceed the number of connections, you can see something similar in/var/log/message:

May 16:13:52 hostname kernel:nf_conntrack:table full, dropping packet

Information, the basic can be determined that FD is not enough to cause. (This information is also available when the server is attacked)


Setup requirements: Let's say we want to set the 200W maximum open file descriptor

1. Modify Nr_open Limit(Use: Able to configure the maximum number of nofile)

Cat/proc/sys/fs/nr_open

Linux kernel 2.6.25 Previously, the macro definition in the kernel is 1024*1024, the largest can only be 100w (1048576), so do not set a larger value, if the Linux kernel is greater than 2.6 25 can set a larger value.

To set the method:

sudo bash-c ' echo 2000000 >/proc/sys/fs/nr_open '

Note: The following restrictions can only be modified if the Nr_open limit is modified. (If the default existing value of Nr_open is higher than our 200w, you can not modify it)

2, open the file descriptor restrictions:Modify limits.conf nofile Soft hard Open File Restrictions (usage: TCP connections)

(1) Temporary entry into force

If you want to modify the temporary effective file descriptor for the current login session, you can use the Ulimit command directly:

Ulimit-shn 2000000

The proper use of the program will not exceed the limit, but restarting the server will fail.
If you want to stay in effect, you can save this content to the startup, sync to:/etc/rc.local file

sudo echo "Ulimit-shn 2000000" >>/etc/rc.local

Note: If you need to get/etc/rc.local to take effect next time, be sure to remember that the file must have Execute permissions: sudo chmod +x/etc/rc.local

The next start will automatically execute this sentence, also can be used normally.

(2) Permanent entry into force

File Location:/etc/security/limits.conf

Find Nofile, if not, add at the end of yourself:

2.6.25 and previous kernel set to 100W:

* Soft Nofile 1000000
* Hard Nofile 1000000

2.6.25 version kernel can be set to 200W:

* Soft Nofile 2000000
* Hard Nofile 2000000

Save this file after setting. (This operation must be restarted before it takes effect, if it cannot be restarted, it will not work, and it is not certain whether the use of/sbin/sysctl-p can take effect directly)

Note: If limits.conf is required to take effect, part of the need to load/lib/security/pam_limits.so to take effect (the default is generally not concerned), if you need attention, you need to add a session at the end of/etc/pam.d/login Required/lib/security/pam_limits.so, but the current version of the kernel should be no problem, you can ignore.

3. Open process limit:Modify Nproc Restrictions in limits.conf (usage: number of processes)

Description: If you have no special requirements for the total number of processes, you can modify this option if you are a high-performance, multi-process server that requires a lot of processes to process.
Ulimit-a can see MAX user processes If the value is larger, you can not set the Nproc item.
Configuration file:/etc/security/limits.d/20-nproc.conf (RHEL 7/centos 7, if rhel6.x/centos6.x file in/etc/security/limits.d/90- nproc.conf)

* Soft Nproc 4096
Root Soft Nproc Unlimited

is root infinity (the actual root user limit is: 255622), and the other non-root user is 4,096 processes.

Description

The hard limit indicates the maximum value that can be set in the soft limit. The soft limit refers to the setting value in effect for the current system. Hard limit values can be reduced by ordinary users. But it can't be increased. The soft limit cannot be set higher than the hard limit. Only the root user can increase the hard limit value.
When adding a file limit description, you can simply double the current value. The example below, if you want to increase the default value of 1024, the best to increase to 2048, if you want to continue to increase, you need to set to 4096.


4, modify the File-max option(Purpose: Number of allocated file handles)

File-max Value: Specifies the maximum number of file handles that can be allocated (you can use the/proc/sys/fs/file-nr file to view the number of file handles and total handles that are currently in use). ）

(1) Temporary entry into force:

File path:/proc/sys/fs/file-max

Cat/proc/sys/fs/file-max

3252210


If you want to modify, directly overwrite the file: (for example, change to 200w)

sudo echo 2000000 >/proc/sys/fs/file-max

Note: If you want to automatically execute the above command every time you start, you can add a command to the system startup configuration file/etc/rc.local: (almost as a permanent entry)

echo 2000000 >/proc/sys/fs/file-max

Or the direct shell is all done:

echo "echo 2000000 >/proc/sys/fs/file-max" >>/etc/rc.local

Note: If you need to get/etc/rc.local to take effect next time, be sure to remember that the file must have Execute permissions: sudo chmod +x/etc/rc.local

(2) Permanent entry into force:

Modify configuration file, file location:/etc/sysctl.conf

Open the configuration file to the very end, and if it is not in the configuration file, you can add it directly:

sudo echo "Fs.file-max = 2000000" >>/etc/sysctl.conf

Configuration file is in effect: sudo/sbin/sysctl-p

5, modify TCP and other related options

Configuration file:/etc/sysctl.conf

Modify Options:

Net.core.somaxconn = 2048
Net.core.rmem_default = 262144
Net.core.wmem_default = 262144
Net.core.rmem_max = 16777216
Net.core.wmem_max = 16777216
Net.core.netdev_max_backlog = 20000
Net.ipv4.tcp_rmem = 4096 4096 16777216
Net.ipv4.tcp_wmem = 4096 4096 16777216
Net.ipv4.tcp_mem = 786432 2097152 3145728
Net.ipv4.tcp_max_syn_backlog = 16384
Net.ipv4.tcp_fin_timeout = 30
Net.ipv4.tcp_keepalive_time = 300
Net.ipv4.tcp_max_tw_buckets = 5000
Net.ipv4.tcp_tw_reuse = 1
Net.ipv4.tcp_tw_recycle = 1
Net.ipv4.tcp_syncookies = 1
Net.ipv4.tcp_max_orphans = 131072
Fs.file-max = 2000000
Fs.inotify.max_user_watches = 16384
Net.netfilter.nf_conntrack_max = 6553500 #本选项在一些版本下无效, you can delete
net.netfilter.nf_conntrack_tcp_timeout_established = 1200 #本选项在一些版本下无效, you can delete

Configuration file is in effect: sudo/sbin/sysctl-p

The above options can also directly modify the memory temp value directly under the/proc/sys/net/directory by using the options similar to echo value >/proc/sys/net/core/wmem_max.

Mainly look at these items:

NET.IPV4.TCP_RMEM is used to configure the size of the read buffer, three values, the first is the minimum value for this read buffer, the third is the maximum, and the middle is the default value. We can modify the size of the read buffer in the program, but not more than the minimum and maximum. To minimize the amount of memory used for each socket, I set the default value of 4096.
NET.IPV4.TCP_WMEM is used to configure the size of the write buffer. Read buffer and write buffer in size, directly affect the socket in the kernel memory footprint;
Net.ipv4.tcp_mem is the amount of memory configured for TCP, in units of pages, not bytes. When the second value is exceeded, TCP enters pressure mode, at which point TCP attempts to stabilize its memory usage and exits pressure mode when it is less than the first value. When the memory occupies more than the third value, TCP refused to allocate the socket, see DMESG, will play a lot of log "Tcp:too many of orphaned sockets";
Net.ipv4.tcp_max_orphans This value should also be set, this value indicates that the system can handle the number of sockets that are not part of any process, when we need to quickly establish a large number of connections, we need to focus on this value. When the number of sockets that do not belong to any process is greater than this value, DMESG will see "Too many of orphaned sockets";
Net.ipv4.tcp_syncookies = 1 means to open syn Cookies. When the SYN wait queue overflow occurs, cookies are enabled to handle, to prevent a small number of SYN attacks, the default is 0, indicating shutdown;
Net.ipv4.tcp_tw_reuse = 1 means to turn on reuse. Allows time-wait sockets to be re used for a new TCP connection, which defaults to 0, indicating shutdown;
Net.ipv4.tcp_tw_recycle = 1 means to open the Time-wait Sockets fast recovery in TCP connection, the default is 0, which means close;
Net.ipv4.tcp_fin_timeout Modify the system? Default timeout time;
The net.ipv4.tcp_max_syn_backlog enters the maximum request queue for the SYN packet. Default 1024. For a heavily loaded server, it is obviously beneficial to increase this value. Can be adjusted to 16384;
Net.ipv4.tcp_keepalive_time = 300 means the frequency at which TCP sends keepalive messages when KeepAlive is enabled. The default is 2 hours, 300 seconds;
Net.ipv4.tcp_max_tw_buckets = 5000 indicates that the system maintains the maximum number of time_wait sockets at the same time, and if this number is exceeded, the time_wait socket is immediately cleared and the warning message is printed. Default is 180000, change to 5000;
Fs.file-max = 2000000 refers to the maximum number of file descriptors that can be opened, and if the system complains: too many file opened, you need to modify this value (this value must be modified with/etc/security/limits.conf to take effect);
fs.inotify.max_user_watches = 16384 Set file system change monitor online. If not full of all kinds of normal situation, also appear tail-f this watch event error no space left on device is this value is not enough;

Note: If it is a client program, for better access to the server program is not card on the port allocation, it is recommended that the client's port (Port_range) a larger range:

Modify File:/etc/sysctl.conf

Net.ipv4.ip_local_port_range = 1024 65535

Configuration effective: Sudo/sbin/sysctl-p

If the client, other file open restrictions can be referenced above to set.

6, some other configuration

(1) Open core file

If in order to observe the program is normal, after the problem generated by the corresponding image file, you can open coredump related operations can be opened: (not necessary, if the online environment, worry about the stability of the impact, you can consider not to open)

Configuration file:/etc/security/limits.conf

To modify a configuration file:

Increase:

* Soft Core 102400
* Hard Core 2048003

The recommended setting is unlimited size:

* Soft Core Unlimited
* Hard Core Unlimited

Then restart the machine to take effect (not sure whether it can use/sbin/sysctl-p), use: Ulimit-a or ulimit-c to view the results, if the program appears stack overflow, etc. will generate coredump files, easy to use GDB and other tracing problems.

(2) Modify other limits.conf configuration

If you want the temporary current session to allow/etc/security/limits.conf to take effect, you can directly use the Ulimit command to modify, in the current sessions on the direct entry into force (exit login or restart failure, in order to permanently effective, you must directly modify the/etc/ security/limits.conf file)

ULIMIT-SHC Unlimited #修改coredump文件大小, after the current session has been modified to take effect, the startup program will directly use this new configuration value
Ulimit-shn 10000000 #修改打开文件数量限制为100W, effective after the current session is modified
Ulimit-shu 4096 #修改当前用户打开进程数量限制为4096个, the current session is directly effective after the modification

#修改完成后使用 Ulimit-a can see the effect of the modification, you need to modify the/etc/security/limits.conf file in effect, and then restart the server to take effect #

optimize Linux kernel parameters to improve server concurrency processing capabilities

The Linux system, after the TCP connection is disconnected, will retain a certain amount of time in the TIME_WAIT state before releasing the port. When there are too many concurrent requests, a large number of time_wait states will be connected, which can not be disconnected in time, and will consume a lot of port resources and server resources. At this time we can optimize the TCP kernel parameters, in time to the TIME_WAIT state of the port clean off.

The method described in this article only causes system resource consumption to be effective for connections with a large number of time_wait states, and if this is not the case, the effect may not be obvious. You can use the netstat command to check the connection state of the TIME_WAIT state, enter the following combination command to view the current TCP connection status and the corresponding number of connections:
#netstat-N | awk '/^tcp/{++s[$NF]} end {for (a in S) print A, s[a]} '
This command outputs a result similar to the following:
Last_ack 16
SYN_RECV 348
Established 70
Fin_wait1 229
Fin_wait2 30
CLOSING 33
Time_wait 18098
We only care about the number of time_wait, where we can see that there are more than 18,000 time_wait, which takes up more than 18,000 ports. To know the number of ports is only 65,535, occupy one less one, will seriously affect the subsequent new connection. In this case, we need to adjust the Linux TCP kernel parameters, so that the system faster release time_wait connection.

Open profile with vim: #vim/etc/sysctl.conf

In this file, add the following lines:
Net.ipv4.tcp_syncookies = 1
Net.ipv4.tcp_tw_reuse = 1
Net.ipv4.tcp_tw_recycle = 1
Net.ipv4.tcp_fin_timeout = 30

Enter the following command for the kernel parameter to take effect: #sysctl-P

Simply explain the meaning of the above parameters:

Net.ipv4.tcp_syncookies = 1
#表示开启SYN Cookies. When the SYN wait queue overflow occurs, cookies are enabled to handle, to prevent a small number of SYN attacks, the default is 0, indicating shutdown;
Net.ipv4.tcp_tw_reuse = 1
#表示开启重用. Allows time-wait sockets to be re used for a new TCP connection, which defaults to 0, indicating shutdown;
Net.ipv4.tcp_tw_recycle = 1
#表示开启TCP连接中TIME-wait Sockets Fast recovery, the default is 0, indicating close;
Net.ipv4.tcp_fin_timeout
#修改系? The default TIMEOUT time.

After this adjustment, in addition to further increase the load capacity of the server, but also to protect against small traffic levels of DOS, CC and SYN attacks.

In addition, if you have a lot of connection number, we can optimize TCP's available port range, further enhance the server's concurrency capabilities. Still in the parameter file above, add the following configuration:
Net.ipv4.tcp_keepalive_time = 1200
Net.ipv4.ip_local_port_range = 10000 65000
Net.ipv4.tcp_max_syn_backlog = 8192
Net.ipv4.tcp_max_tw_buckets = 5000
#这几个参数, it is recommended that you only open on a server with very large traffic, which can have a significant effect. General traffic on a small server, there is no need to set these parameters.

Net.ipv4.tcp_keepalive_time = 1200
#表示当keepalive起用的时候, the frequency with which TCP sends keepalive messages. The default is 2 hours, and 20 minutes instead.
Net.ipv4.ip_local_port_range = 10000 65000
#表示用于向外连接的端口范围. Small by default: 32768 to 61000, 10000 to 65000. (Note: Do not set the minimum value too low, otherwise it may take off the normal port!) ）
Net.ipv4.tcp_max_syn_backlog = 8192
#表示SYN队列的长度, the default is 1024, which increases the queue length to 8192, and can accommodate more network connections waiting for connections.
Net.ipv4.tcp_max_tw_buckets = 6000
#表示系统同时保持TIME_WAIT的最大数量, if this number is exceeded, time_wait is immediately cleared and the warning message is printed. The default is 180000, and 6000 is changed. For Apache, Nginx and other servers, the parameters on a few lines can well reduce the number of time_wait sockets, but for Squid, the effect is not. This parameter can control the maximum number of time_wait, avoid squid server by a large number of time_wait dragged to death.

Kernel Other TCP parameters Description:
Net.ipv4.tcp_max_syn_backlog = 65536
#记录的那些尚未收到客户端确认信息的连接请求的最大值. For systems with 128M of memory, the default value is 1024, and the small memory system is 128.
Net.core.netdev_max_backlog = 32768
#每个网络接口接收数据包的速率比内核处理这些包的速率快时 the maximum number of packets that are allowed to be sent to the queue.
Net.core.somaxconn = 32768
#web应用中listen函数的backlog默认会给我们内核参数的net. Core.somaxconn is limited to 128, and Nginx defines ngx_listen_backlog defaults to 511, so it is necessary to adjust this value.

Net.core.wmem_default = 8388608
Net.core.rmem_default = 8388608
Net.core.rmem_max = 16777216 #最大socket读buffer, reference to the optimized value: 873200
Net.core.wmem_max = 16777216 #最大socket写buffer, reference to the optimized value: 873200
Net.ipv4.tcp_timestsmps = 0
#时间戳可以避免序列号的卷绕. A 1Gbps link is sure to encounter a previously used serial number. The timestamp allows the kernel to accept this "exception" packet. We need to turn it off.
Net.ipv4.tcp_synack_retries = 2
#为了打开对端的连接, the kernel needs to send a SYN with an ACK that responds to a previous syn. The second handshake in the so-called three handshake. This setting determines the number of Syn+ack packets sent before the kernel discards the connection.
Net.ipv4.tcp_syn_retries = 2
#在内核放弃建立连接之前发送SYN包的数量.
#net. Ipv4.tcp_tw_len = 1
Net.ipv4.tcp_tw_reuse = 1
# Open Reuse. Allows time-wait sockets to be reconnected to a new TCP connection.

Net.ipv4.tcp_wmem = 8192 436600 873200
# TCP Write buffer, can refer to the optimization value: 8192 436600 873200
Net.ipv4.tcp_rmem = 32768 436600 873200
# TCP Read buffer, can refer to the optimization value: 32768 436600 873200
Net.ipv4.tcp_mem = 94500000 91500000 92700000
# also has 3 values, meaning:
NET.IPV4.TCP_MEM[0]: Under this value, TCP has no memory pressure.
NET.IPV4.TCP_MEM[1]: Under this value, enter the memory pressure phase.
NET.IPV4.TCP_MEM[2]: higher than this value, TCP refuses to allocate socket.
The above memory units are pages, not bytes. The optimized values to refer to are: 786432 1048576 1572864

Net.ipv4.tcp_max_orphans = 3276800
#系统中最多有多少个TCP套接字不被关联到任何一个用户文件句柄上.
If this number is exceeded, the connection is immediately reset and a warning message is printed.
This limit is only to prevent simple Dos attacks, not to rely too much on it or artificially reduce this value,
This value should be added (if memory is added).
Net.ipv4.tcp_fin_timeout = 30
#如果套接字由本端要求关闭, this parameter determines how long it remains in the fin-wait-2 state. The right end can be an error and never close the connection, or even accidentally machine. The default value is 60 seconds. 2.2 The normal value of the kernel is 180 seconds, you can press this setting, but keep in mind that even if your machine is a lightweight Web server, there is a risk of memory overflow because of a lot of dead sockets, fin-wait-2 is less dangerous than fin-wait-1, Because it can only eat 1.5K of memory, but they have a longer lifetime.

After this optimized configuration, your server's TCP concurrency is significantly improved. The above configuration is for reference only, for production environment, please according to their own actual situation.