Haproxy (vi) Performance optimization

TCP counter listenoverflows a large increase

View Method

Cat/proc/net/netstat | awk '/tcpext/{print $21,$22} '

reason

The queue length of the system call listen function (int listen (int sockfd, int backlog) is determined by min (backlog, kernel parameter net.core.somaxconn), corresponding to the listen of the socket When the queue is full, the listenoverflows counter adds 1 when a new connection is added.

Solution

Adjust system Parameters Net.core.somaxconn = 65535 #默认为128

Cat 65535 >/proc/sys/net/core/somaxconn

Haproxy Exhaustion CPU Problem

phenomena

Haproxy bind 4 Core CPU, 8 core CPU, 16 core CPU, when the pressure is measured, will appear CPU 100% is used situation.

Tracing Process

1. Through the Mpstat-p all 1 command to see where the CPU is used, you can see most of the CPU used in the SYS state.

2. View the Haproxy process through Strace-p pid-c the system call takes up the most CPU and sees the CPU used on the Connect system call.

3. Haproxy Connect back end will use Connect system call, view Haproxy configuration, found that haproxy connected to the back end of only two backend, that is, this situation: Vegeta (40 pressure machine)-"haproxy-" Httptest (2 haproxy back end), where the haproxy connected to the back end can only have 63,000 ports or so, a large number of connections backlog in the Haproxy machine, resulting in the CPU is depleted.

 

Solution

A large number of haproxy to increase the back end, to solve the haproxy the bottleneck of too little rear end.

Network card Drop Packet

phenomena

When the pressure test, Ifconfig found that the NIC Eth0 RX appeared droppped.

Solution

1. Adjust the ring buffer of network card

View: Ethtool-g eth0

Set ring buffer to upper limit: ethtool-g eth0 rx 4096;ethtool-g eth0 TX 4096

2. Transfer network card team cache queue

Net.core.netdev_max_backlog = 2000000 #默认为1000, when kernel parameters are configured, there is a specific explanation

Latest stable version Haproxy performance

system with version 1.5.18

compile Haproxy Latest stable version (1.7.5)

wget http://www.haproxy.org/download/1.7/src/haproxy-1.7.5.tar.gz
TAR-XZVF haproxy-1.7.5.tar.gz
CD haproxy-1.7.5/make
target=linux2628 use_cpu_affinity=1

use of resources during pressure measurement

Mem:

Latest New Edition: 10G

System with version: 102G, due to insufficient memory, the use of swap,

Cause CPU iowait accounted for a large increase, resulting in CPU idle 0.

CPU Idle:

Latest edition: 35%

System with version: 0%

Conclusions

Under the same pressure, the new version of Haproxy uses a large amount of memory reduction.

Environment construction

Haproxy

Machine

1 I1 machines with CPU and 128G memory

System Version

CentOS 7.2

Installation system with Haproxy:yum install Haproxy

sysctl.conf Configuration

Maximum number of file handles allowed in the ### system Fs.file-max = 12553500 ### The maximum number of file handles allowed by a single process Fs.nr_open = 12453500 ### The shared memory that the kernel allows to use is large Controls the maxi Mum number of shared memory segments, in pages Kernel.shmall = 4294967296 # # #单个共享内存段的最大值 Controls the maximum shared Segme NT size, in bytes Kernel.shmmax = 68719476736 ### The maximum number of messages in message queues in the kernel Controls the maximum size of a message, in bytes kernel.m Sgmax = 65536 ### System Rescue Tool KERNEL.SYSRQ = 0 ### The maximum number of packets that are allowed to be sent to the cache queue when the rate of packets received per network interface is faster than the kernel processes these packets net.core.netdev_max_backlog = 2000000 ### Default TCP data Receive window size (bytes) Net.core.rmem_default = 699040 ### Largest TCP data Receive window (byte) Net.core.rmem_max = 50331648 ### default T CP Data Send window size (bytes) Net.core.wmem_default = 131072 ### maximum TCP data Send window (bytes) Net.core.wmem_max = 33554432 ### Defines the length of the maximum listener queue for each port in the system, which is a global parameter Net.core.somaxconn = 65535 ### TCP/UDP protocol allows local port number Net.ipv4.ip_local_port_range = 15000 65000 Net.ipv4.ip_nonlocal_bind = 1 ### The time for TCP to remain in Fin-wait-2 state (in seconds) Net.ipv4.tcp_fin_timeout = 7 for a socket connection that is disconnected from the port (s) (### The time interval (in seconds) TCP sends keepalive probe messages to confirm that the TCP connection is valid Net.ipv4.tcp_keePalive_time = Net.ipv4.tcp_max_orphans = 3276800 ### The maximum number of connections that can be saved in the queue for connection requests that have not yet been validated Net.ipv4.tcp_max_syn_backlog = 655360 net.ipv4.tcp_max_tw_buckets = 6000000 ### determines how the TCP stack should reflect memory usage, with each value in units of memory pages (usually 4KB) ### the first value is the lower bound of memory usage
; The second value is the upper bound of the pressure applied to the buffer using the memory pressure mode; The third value is the upper limit for memory usage.
Net.ipv4.tcp_mem = 94500000 915000000 927000000 ### defines the memory used by the socket for automatic tuning. ### The first value is the minimum number of bytes allocated for the socket receive buffer; ### the second value is the default value (which is overwritten by Rmem_default), and the buffer can grow to this value if the system load is not heavy; ###
The third value is the maximum number of bytes in the Receive buffer space (the value is overwritten by Rmem_max) Net.ipv4.tcp_rmem = 32768 699040 50331648 ### defines the memory used by the socket for automatic tuning. ### The first value is the minimum number of bytes allocated for the socket send buffer; ### The second value is the default value (which is overwritten by Wmem_default), and the buffer can grow to this value if the system load is not heavy; ###
The third value is the maximum number of bytes in the Send buffer space (this value is overwritten by Wmem_max) Net.ipv4.tcp_wmem = 32768 131072 33554432 net.ipv4.tcp_slow_start_after_idle = 0 Net.ipv4.tcp_synack_retries = 2 ### Indicates whether the TCP synchronization label (Syncookie) is turned on, and the Sync tab prevents a socket from overloading when there are too many attempts to connect ### kernel must be open config_syn_
Cookies are compiled, net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_syn_retries = 2 ### indicates a quick recycle of time-wait sockets on a TCP connection, the default is 0, which means shutdown Net.ipv4.tcp_tw_recycle = 1 ### allows the TIme-wait sockets is re used for a new TCP connection, which defaults to 0 to turn off Net.ipv4.tcp_tw_reuse = 1 ### enable the Windows scaling defined by RFC 1323 to support TCP windows over 64KB. This value must be enabled (1 is enabled) and the ### TCP window will not take effect until both sides of the 1GB,TCP connection are enabled, the default is 1 net.ipv4.tcp_window_scaling = 1 ### maximum use of physical memory Vm.swappiness = 0

Edit/etc/security/limits.conf Add the following content

* Soft Nofile 10000000
* Hard nofile 10000000
root soft nofile 10000000
root hard Nofile 10000000

haproxy Configuration

global log/dev/log local0 log/dev/log local1 notice chroot/var/lib/haproxy stats socket/run/haproxy /admin.sock mode 660 level admin stats Timeout 600s User haproxy Group Haproxy daemon #tune. Ssl.default-dh-param 2048 MAXC Onn 4000000 nbproc cpu-map 1 1 cpu-map 2 2 cpu-map 3 3 cpu-map 4 4 cpu-map 5 5 cpu-map 6 6 Cpu-map 7 7 Cpu-map 8 8 cpu- Map 9 9 Cpu-map cpu-map cpu-map cpu-map cpu-map to Cpu-map Cpu-map (CP) (MB). U-map cpu-map cpu-map cpu-map cpu-map cpu-map Stats (a) (A/bind-process) 
L mode TCP option Tcplog option dontlognull maxconn 4000000 Timeout Connect 5000 Timeout client 60000 Timeout server 60000 # Template Customization frontend http-in bind:8200 stats enable mode HTTP option Httplog stats auth admin:admin stats u Ri/haproxy_stats Listen port-30000 bind:30000 mode TCP option Tcplog balance Roundrobin server staging1 .... 

Haproxy Back-end service

Machine

10, 8 CPU, 32G memory 3u8 Machine

System Version

CentOS 7.2

program

Shorthand for an API server

Machine for pressure measurement

Machine

80, 8/32 cpu,32/128g memory machine, machine configuration is not uniform, so use 80 machines to do the pressure test, to solve the pressure test end performance, the core parameters of the machine uneven problem.

program

Vegeta Https://github.com/tsenart/vegeta

Summarize

Compound anticipation of pressure measurement effect

The pressure test effect basically conforms to the anticipation, the final establish state connection achieves 696W, occupies 10G memory, the CPU uses 66% (temporarily does not have the SSL demand, therefore does not have the pressure to test the SSL uninstall), detailed as shown:

Key knowledge points

TCP connection, Srcip:srcport-"dstip:dstport srcport up to 63k, when concurrent traffic exceeds 63k, the application process consumes a lot of CPU." Note that the machine is also subject to this limitation.

Reference articles:

Https://www.cnblogs.com/276815076/p/8004039.html

https://mp.weixin.qq.com/s?__biz=MzAwMDU1MTE1OQ==&mid=2653548665&idx=1&sn= C0d489dcacd0c7c05e579ed74c9050c9&scene=21#wechat_redirect