Linux Server high concurrency practice experience

As a wild programmer who left early as a master, the business can also reach the level of bluffing, but the rise to the performance level that is the mishap.

Real pie in the sky, the company assigned a test performance task, really feel my sky stars are bright.

High concurrency major limiting factors: CPU, network traffic, memory, system configuration

Cpu

Using top to see CPU utilization, press 1 to see how each CPU thread works; This shows the CPU idle, utilization, soft interrupt status

If a CPU thread usage often reaches 100%, then the CPU becomes a bottleneck, usually in order to achieve high concurrency, the larger load service program will bind the CPU itself, so that its tasks are allocated to multiple CPU threads to ensure stable operation of the program

How to bind the CPU:

The Ncpuindex represents the CPU number, starting with 0.
cpu_set_t Mask;
Cpu_zero (&mask);
Cpu_set (Ncpuindex, &mask);
Sched_setaffinity (M_hthread, sizeof (mask), &mask);

Network card traffic

For streaming media server, the network card is absolutely the main bottleneck, even the million Gigabit network card, the face of 1Mbps code stream, concurrency is only 10,000;

Network card traffic mainly through the dstat-n to specify multiple network cards to monitor, in the single-machine test process, it is necessary to CCIE support:

(1) First bind multiple network cards with Bond (MOD4), but the corresponding gigabit switches also need to be consistent with the server side.

Here is a knowledge point is "multi-nic binding seven modes", good switch support more network port load mode, will make each network port traffic basically balanced.

If the configuration error, in the process of DSTAT statistics will be found that some network card traffic is full, but some do not have traffic, resulting in a large number of test instances dropped line;

Dstat function is very full, the CPU, memory can be specified, such as read column high traffic, it indicates that the program read and write disk operations more frequently.

(2) In addition, the NIC CPU interrupt is also best to confirm:

To view the current NIC's terminal number: cat/proc/interrupts | grep eth1

To view the CPU allocated by the current NIC (98 is the result of the first step): Cat/proc/irq/98/smp_affinity_list

Allocate a more idle CPU to the network card: Echo >/proc/irq/98/smp_affinity_list

Memory

can use top, free-m and so on to view, anyway consumes too much is the problem of the program. Each value of free can also see how the program works.

[Email protected] ~]# free-m
Total used free shared buffers Cached
mem:31993 1596 30397 0 12 22
-/+ buffers/cache:1562 30431
swap:15999 7 15992

otal--Total Physical Memory
used--already uses memory, this value is generally larger because this value includes the memory used by the cache+ application
free--memory that is not fully used
shared--Application Shared Memory
buffers--cache, mainly used for directory aspects, Inode value, etc. (LS Large directory can see this value increase)
cached--cache, for files that have been opened

Note
Total=used+free
Used=buffers+cached (maybe add shared also)

The second line describes the memory usage of the application:
The previous value indicates the memory size used by the-buffers/cache--application, used minus the cached value
The latter value indicates the amount of memory +buffers/cache--all available to the application, free plus the cached value
Note
-buffers/cache=used-buffers-cached
+buffers/cache=free+buffers+cached

The third line represents the use of swap:
used--already in use
free--not used

System parameters

Before you played a virtual machine,

File descriptor The general configuration of the horse is 65535 and no problem is found.

I don't think there's anything wrong with a socket in time_wait state.

But these are very important for high-concurrency servers.

Frequent HTTP services will create a large number of short connections, there will be a large number of time_wait in the 2ML timeout period, the occupancy descriptor,

If you happen to configure the system maximum descriptor is also very small, the performance of course will not go.

System configuration is mainly modified:/etc/sysctl.conf files, modified after sysctl-p to update

Net.ipv4.tcp_max_tw_buckets = 6000
The number of timewait, by default, is 180000.
Net.ipv4.ip_local_port_range = 1024 65000
Allows the system to open a range of ports.
Net.ipv4.tcp_tw_recycle = 1
Enable Timewait Quick Recycle.
Net.ipv4.tcp_tw_reuse = 1
Turn on reuse. Allows time-wait sockets to be re-used for new TCP connections.
Net.ipv4.tcp_syncookies = 1
Turn on SYN cookies to enable cookies to be processed when a SYN wait queue overflow occurs.
Net.core.somaxconn = 262144
The BACKLOG of LISTEN functions in Web applications restricts the net.core.somaxconn of our kernel parameters to 128, and the Nginx-defined ngx_listen_backlog defaults to 511, so it is necessary to adjust this value.
Net.core.netdev_max_backlog = 262144
The maximum number of packets that are allowed to be sent to the queue when each network interface receives a packet at a rate that is faster than the rate at which the kernel processes these packets.
Net.ipv4.tcp_max_orphans = 262144
The maximum number of TCP sockets in the system are not associated with any one of the user file handles. If this number is exceeded, the orphan connection is immediately reset and a warning message is printed. This limitation is only to prevent a simple Dos attack, not to rely too much on it or artificially reduce the value, but should increase this value (if the memory is increased).
Net.ipv4.tcp_max_syn_backlog = 262144
Record the maximum number of connection requests that have not received the client acknowledgment information. For systems with 128M of memory, the default value is 1024, and the small memory system is 128.
Net.ipv4.tcp_timestamps = 0
Timestamps can prevent the winding of serial numbers. A 1Gbps link will definitely encounter a previously used serial number. Timestamps allow the kernel to accept this "exception" packet. You need to turn it off here.
Net.ipv4.tcp_synack_retries = 1
In order to open the connection to the end, the kernel sends a SYN and comes with an ACK that responds to the previous syn. The second handshake in the so-called three-time handshake. This setting determines the number of Syn+ack packets sent before the kernel abandons the connection.
Net.ipv4.tcp_syn_retries = 1
The number of SYN packets sent before the kernel abandons the connection.
Net.ipv4.tcp_fin_timeout = 1
If the socket is closed by the local side, this parameter determines how long it remains in the fin-wait-2 state. The peer can make an error and never shut down the connection, or even accidentally become a machine. The default value is 60 seconds. 2.2 The normal value of the kernel is 180 seconds, 3 You can press this setting, but remember that even if your machine is a light-load Web server, there is a large number of dead sockets and memory overflow risk, fin-wait-2 is less dangerous than fin-wait-1, Because it can only eat up to 1.5K of memory, but they have a longer lifetime.
Net.ipv4.tcp_keepalive_time = 30
When KeepAlive is employed, the frequency at which TCP sends keepalive messages. The default is 2 hours.

Configuration of the application itself

For example Nginx, preferably according to the number of CPU threads to configure workers, to open Epoll mode, to open sendfile and so on.

Linux Server high concurrency practice experience