Restrictions on the maximum number of concurrent socket connections in Linux

Restrictions on the maximum number of concurrent socket connections in Linux 1. modify the number of files that can be opened by a user process on the Linux platform, regardless of the client program or the server program, when processing highly concurrent TCP connections,
The maximum number of concurrent jobs is limited by the number of files that the system can open at the same time for a single process (this is because the system
Create a socket handle for each TCP connection, and each socket handle is also a file handle ).
You can use the ulimit command to view the number of files allowed by the current user process: www.2cto.com [Alibaba G @ as4 ~] $ Ulimit-n1024 indicates that each process of the current user can open up to 1024 files simultaneously.
Standard input, standard output, standard error, server listening socket,
The number of files that can be used for client socket connection is
Only about 1024-10 = 1014. That is to say, by default, Linux-based communication programs are allowed at most
1014 concurrent TCP connections at the same time. To support a higher number of TCP concurrent connections, you must modify
Soft limit and hardlimit ). Soft restrictions
Linux further limits the number of files simultaneously opened by users within the limits of the current system;
The maximum number of files that can be opened simultaneously by the system based on the system hardware resources (mainly the system memory.
Generally, the soft limit is less than or equal to the hard limit.
The simplest way to modify the preceding limits is to use the ulimit command: [Alibaba G @ as4 ~] $ Ulimit-n www.2cto.com in the preceding command, specify the maximum number of files allowed to be opened by a single process. If the system echo
Similar to "Operation notpermitted", it indicates that the above restriction modification failed.
Because the value specified in exceeds the Linux system's soft or hard limit on the number of files opened by the user.
Therefore, you need to modify the Linux system's soft and hard limits on the number of opened files.
Step 1: Modify/etc/security/limits. conf file. Add the following lines to the file: CMDG soft nofile 10240 CMDG hard nofile 10240 where CMDG specifies the number of files opened by the user to be modified, '*' can be used to modify the restrictions of all users;
Soft or hard: Specifies whether to modify the soft limit or hard limit; 10240 specifies the new limit value to be modified,
That is, the maximum number of opened files (note that the soft limit value must be smaller than or equal to the hard limit ). Save the modified file.
Step 2: Modify/etc/pam. d/login file. Add the following lines to the file: session required/lib/security/pam_limits.so www.2cto.com. This tells Linux to call the pam_limits.so module to set the system
The maximum number of resources that a user can use (including the maximum number of files that a user can open ),
The pam_limits.so module reads the configuration from the/etc/security/limits. conf file to set these limits.
Save the modified file.
Step 3: Check the maximum number of opened files in Linux. Run the following command: [CMDG @ as4 ~] $ Cat/proc/sys/fs/file-max12158 this indicates that this Linux system allows up to 12158 files to be opened simultaneously (that is, containing the total number of files opened by all users,
It is a Linux system-level hard limit. The limit on the number of files opened by all users should not exceed this value. Usually this system level
The hard limit is the optimal maximum number of files that can be opened at the same time based on the hardware resources of the system when the Linux system is started,
If there is no special need, do not modify this limit unless you want to set a value that exceeds this limit for the number of user-level open files.

To modify this hard limit, modify/etc/rc. add the following lines to the local script: echo 22158>/proc/sys/fs/file-max. This allows Linux to forcibly set the number of files opened at the system level to 22158 after startup. Save the modified file.
After completing the preceding steps, restart the system. Generally, the Linux system allows
The maximum number of opened files is set to a specified value. If you use the ulimit-n command to view the limit on the number of files that can be opened after the restart
It is still lower than the maximum value set in the preceding step, probably because the ulimit-n command is used in the user login script/etc/profile.
The number of files that the user can open at the same time has been limited. Because you can use ulimit-n to modify the system
When the maximum number limit is reached, the new value can only be smaller than or equal to the value set in the previous ulimit-n.
The limit value is impossible.
Therefore, if the above problem exists, you can only open the/etc/profile script file,
Check whether ulimit-n is used in the file to limit the maximum number of files that can be opened at the same time. If yes,
Delete this line of command, or change the set value to the appropriate value, and then save the file. You can log out and log on to the system again. Through the above steps, www.2cto.com removes the system restrictions on the number of opened files for the communication handler that supports High-concurrency TCP connection processing. 2. When you modify the network kernel's restrictions on TCP connections and write a client communication handler that supports High-concurrency TCP connections on Linux, you may find that although the system has been removed
Limit on the number of files opened by the user at the same time. However, when the number of concurrent TCP connections increases to a certain number
Establish a new TCP connection. There are many reasons for this.
The first reason may be that the Linux Network kernel has restrictions on the range of local port numbers. At this time, further analysis of why not
When a TCP connection is established, the problem occurs when the connect () call returns an error. Check that the system error message is "Can't assign requestedaddress ". At the same time, if tcpdump is used to monitor the network at this time, the client will find that there is no TCP connection at all
The network traffic of the SYN packet. These situations indicate that there are restrictions on the local Linux kernel.
Actually, the root cause of the problem
The range of the local port number corresponding to all client TCP connections in the Linux Kernel TCP/IP implementation module.
(For example, the kernel limits the local port number range to 1024 ~ Between 32768 ). When the system is at one time
When too many TCP client connections exist, each TCP client connection occupies a unique local port.
(This port number is in the range of the local port number of the system.) If the existing TCP client connection has filled up all the local port numbers,
In this case, you cannot allocate a local port number for the new TCP client connection. Therefore, the system will ()
Failed to return in the call, and set the error message to "Can't assignrequested address ".
For these controls
For the logic, you can view the Linux kernel source code. Taking the linux2.6 kernel as an example, you can view the following functions in the tcp_ipv4.c file: static int tcp_v4_hash_connect (struct sock * sk). Please note the access control of the variable sysctl. Variable sysctl_local_port_range
Is in tcp. in the c file, set the following function: void _ init tcp_init (void) the default local port number range set during kernel compilation may be too small, so you need to modify the local port range limit. Step 1: Modify the/etc/sysctl. conf file and add the following lines to the file: net. ipv4.ip _ local_port_range = 1024 65000. This indicates that the system sets the local port range to 1024 ~ In the range of 65000. Note that the minimum value of the local port range
The value must be greater than or equal to 1024, and the maximum value of the port range should be less than or equal to 65535. Save the modified file. Step 2: run the sysctl command: [mongog @ as4 ~] $ Sysctl-p if no error message is displayed, the new local port range is successfully set. If you set the port range,
In theory, a single process can establish a maximum of 60000 TCP client connections at the same time. Www.2cto.com
The second reason is that the Linux Network kernel's IP_TABLE firewall cannot establish a TCP connection.
The number of connections is limited. At this time, the program will be blocked in the connect () call, as if it is dead. If tcpdump is used to monitor the network,
It will also find the network traffic of the client sending SYN packets when there is no TCP connection at all. Because IP_TABLE Firewall
The status of each TCP connection is tracked. The tracing information is stored in the conntrackdatabase in the kernel memory,
The size of this database is limited. when too many TCP connections exist in the system, the database capacity is insufficient and IP_TABLE cannot be
The new TCP connection establishes the trace information, so it is blocked in the connect () call. In this case, you must modify the kernel-to-maximum trace.
The method is similar to modifying the local port number range of the kernel:
Step 1: Modify/etc/sysctl. conf file, add the following lines to the file: net. ipv4.ip _ conntrack_max = 10240 this indicates that the system sets the maximum number of TCP connections for tracking to 10240. Please note that this limit value should be as small as possible,
To reduce the usage of kernel memory.
Step 2: run the sysctl command: [mongog @ as4 ~] $ Sysctl-p if no error message is displayed, it indicates that the system has successfully modified the maximum number of TCP connections for the new trail.
If the preceding parameters are used, a single process can establish a maximum of 10000 TCP client connections at the same time.
3. When you write a high-concurrency TCP connection application on Linux using programming technology that supports High-concurrency network I/O, appropriate network I/O technology and I/O event dispatch mechanisms must be used. Available I/O technologies include synchronous I/O, non-blocking synchronous I/O (also called reactive I/O), and asynchronous I/O. In the case of high TCP concurrency,
If synchronous I/O is used, this will seriously block the running of the program unless a thread is created for the I/O of each TCP connection.

However, too many threads cause huge overhead for Thread Scheduling by the system. Therefore, in the case of high TCP concurrency
Synchronous I/O is not desirable. In this case, you can consider using non-blocking synchronous I/O or asynchronous I/O. Non-blocking I/O synchronization techniques include the use of select (), poll (), epoll and other mechanisms. The asynchronous I/O technology is to use AIO. Www.2cto.com
From the perspective of the I/O event allocation mechanism, it is inappropriate to use select () because it supports a limited number of concurrent connections (usually within 1024 ).
If performance is considered, poll () is not suitable. Although it supports a high number of TCP concurrency
The "Round Robin" mechanism, when the number of concurrency is high, its operation efficiency is quite low, and there may be uneven allocation of I/O events, resulting in some TCP
I/O on the connection is "hunger. If epoll or AIO is used, there is no such problem (the AIO technology of the early Linux Kernel
Implementation is achieved by creating a thread for each I/O request in the kernel. This implementation mechanism is implemented in the case of high-concurrency TCP connections.
In fact, there are also serious performance problems. However, the implementation of AIO has been improved in the latest Linux kernel ).
To sum up, when developing a Linux application that supports High-concurrency TCP connections, epoll or AIO technology should be used as much as possible to achieve concurrency.
I/O control over TCP connections, which provides effective I/O assurance for the support of High-concurrency TCP connections. Kernel Parameter sysctl. conf optimization/etc/sysctl. conf is used to control the configuration files of linux networks. For Network-dependent programs (such as web servers and cache servers)
It is very important that RHEL provides the best adjustments by default. Recommended configuration (put the original/etc/sysctl. clear the conf content and copy the following content): net. ipv4.ip _ local_port_range = 1024 65536net. core. rmem_max = 16777216net. core. wmem_max = 16777216net. ipv4.tcp _ rmem = 4096 87380 16777216net. ipv4.tcp _ wmem = 4096 65536 16777216net. ipv4.tcp _ fin_timeout = 10net. ipv4.tcp _ tw_recycle = 1net. ipv4.tcp _ timestamps = 0net. ipv4.tcp _ window_scaling = 0net. ipv4.tcp _ sack = 0 www.2cto.com net. core. netdev_max_backlog = 30000net. ipv4.t Cp_no_metrics_save = 1net. core. somaxconn = 262144net. ipv4.tcp _ syncookies = 0net. ipv4.tcp _ max_orphans = 262144net. ipv4.tcp _ max_syn_backlog = 262144net. ipv4.tcp _ synack_retries = 2net. ipv4.tcp _ syn_retries = 2. For details about this configuration, refer to the recommended configuration of the cache server varnish and the recommended configuration of SunOne server system optimization. Varnish tuning recommendation configuration address: http://varnish.projects.linpro.no/wiki/Performance but varnish Recommended configuration is problematic, the actual operation shows "net. ipv4.tcp _ fin_timeout = 3" configuration
This will often cause the page to be unable to be opened. When a netizen uses the Internet Explorer 6, after accessing the website for a period of time, all webpages will
Cannot open. Restart the browser and the browser will be normal. It may be the speed of the Internet outside China. Our national conditions determine the need.
Adjust "net. ipv4.tcp _ fin_timeout = 10". In 10 s, everything is normal (actual operation conclusion ). After the modification, run the/sbin/sysctl-p/etc/sysctl. conf/sbin/sysctl-w net. Route 4.route. flush = 1 command. For the sake of insurance, you can also reboot the system. Adjust the number of files: After optimizing the network in www.2cto.com linux, you must increase the number of files allowed to be opened in order to support high concurrency. By default, 1024 is far from enough. Run the command: Shell code echo ulimit-HSn 65536>/etc/rc. localecho ulimit-HSn 65536>/root/. bash_profileulimit-HSn 65536
Author guowake