Tengine/nginx performance optimization and its talk

Google, degree Niang search Nginx optimization, can search out a lot of articles, tens of thousands of concurrent, 100,000 concurrency, looking like it is really so.

From the use of tengine process, the optimization of Tengine/nginx, I personally think that tengine optimization is inseparable from the use of its environment and deployment structure, the significance of the optimization of Tengine is not small, and each company's business is different, so optimization is not a simple thing. Even so, I summed up the next, from the following several convenient to discuss the performance optimization of the lower Tengine/nginx. This paper mainly Tengine, nginx most of them are applicable.

1. Network

Bandwidth

A need to support 10,000 concurrent and average page content in the case of 100KB, you need about 1G of bandwidth, if the bandwidth is only 100M, it is impossible to support such a scenario anyway. Better bandwidth can support higher concurrency

Routing

There are many IDC operators, the smaller operators in the route optimization is not good, for example, when the IDC test, most operations use ping to test the network quality of the room, this test is actually point-to test, the middle of the large network of the situation is not tested. When testing with MTR, you can see the delay and routing of each hop, and many times you will find a floating route. In this case, the quality of the network must be bad. Preferably through the MTR to the computer room to provide a continuous test of the IP for a week, the path is relatively stable when such a room is worth choosing.

2. Hardware

CPU

In general, as a Tengine/nginx application, CPU frequency I think the performance impact is still small, but the core number for Tengine/nginx is really useful, more core number plus HT technology, you can let tengine/ Nginx runs more workers, which increases the load capacity of the Tengine/nginx

Memory

The capacity of memory directly affects the number of connections Tengine/nginx can host (mainly estab connections). In addition, a large amount of memory in conjunction with TMPFS for reasonable use, can be used as a variety of Tengine/nginx temp directory and various cache directories, reduce the consumption of disk IO

Network card

At present, the Internet company Dell's server should be used the most, and in the absence of special needs of users, the configuration of the network card is usually Broadcom bcm57xx, this network card is cheaper all in the processing of small packets, bandwidth utilization is not high, and the traffic will appear a large number of packet loss. The Intel x350i QP Network card is recommended on Tengine/nginx machines, and more performance control can be made on the NIC after using the latest NIC driver (X350 default RSS queue up to 8)

Disk

The IO capabilities of the disk personally believe that the proportion of hardware is not high, when taking Tengine/nginx as a dedicated cache server, when using a SATA SSD or PCIE-SSD card can be used as a directory path cache storage. General Intel S3710 800G of about 7000, the same capacity Baoqun 800G PCIE-SSD about 1.5W.

3. System

TCP/IP protocol stack

close_wait and Time_wait

These two connection states are the most concerned about the Web server is also the most let everyone headache, although a lot of blog said now server memory is large, this connection reached tens of thousands of, hundreds of thousands of when the memory consumption is not a problem to ignore, but in some scenarios may have a great impact on the application. For example, when the close-wait state occurs in our business, when the database has a long transaction execution, it causes the resin to have a close-wait connection, and when the number of such connections is higher than 10, the application will open slowly until the application is opened. So it needs to be optimized. Try to avoid having too many of these two connections in the system.

When Tengine/nginx reverse proxy back-end server, tengine on the server will appear time-wait (after the back-end server disconnects the resulting state) and close-wait (the client disconnects the resulting state) of the connection, Generally in order to reduce the tengine/nginx and back-end server communication caused by the time-wait connection, the main need for the following configuration, for example:

Configure a

Upstream Example {
Server 192.168.0.1:8080;
KeepAlive 4;
}
server{
Listen backlog=2048;
server_name www.example.com;
Location =/favicon.ico {
Error_page 404 = 200;
Log_not_found off;
Access_log off;
}
Location =/robots.txt {
Error_page 404 = 200;
Log_not_found off;
Access_log off;
}
Location/{
Include proxy_opt.conf;
Proxy_http_version 1.1;
Proxy_set_header Connection "";
Proxy_pass http://example;
}
}

Configuration Two

server{
Listen backlog=2048;
server_name www.example.com;
Location =/favicon.ico {
Error_page 404 = 200;
Log_not_found off;
Access_log off;
}
Location =/robots.txt {
Error_page 404 = 200;
Log_not_found off;
Access_log off;
}
Location/{
Include proxy_opt.conf;
Proxy_pass http://192.168.0.1:8080;
}
}

Two configurations, can achieve www.example.com access, the difference is that configuration one because of the use of upstream and configured in the upstream KeepAlive connection, when the pressure is large, the close-wait between the Nginx and the backend server and time-wait will be very small, because To have a long connection. And the configuration of the second is direct proxy_pass, in the high pressure when the number of close-wait and time-wait will be more, in this case, easy to create orphan connection, orphan connection generally takes up about 64K of memory and is non-exchangeable memory (can refer to the kernel document). In Tengine, long connections to upstream are not only configurable in upstream, but long connection timeout parameters are also supported. (It is estimated that the optimization of the TCP/IP protocol stack can reduce the number of connections between close-wait and time-wait, really?) You'll find out if you try. ）

In addition to enabling upstream on nginx to reduce the number of connections between close-wait and time-wait, there are 2 different ways:
A) Change the value of Net.ipv4.tcp_max_tw_buckets to 0 in/etc/sysctl.conf to quickly release the Time-wait connection (this method is more violent, but it works better when the system load is relatively high, somewhat like the haproxy Option Forceclose)

b) Edit kernel in Include/net/tcp.h
Will
#define Tcp_timewait_len (60*HZ)
Revision changed to
#define Tcp_timewait_len (5*HZ)
Recompile the kernel

4. Tengine itself

choice of installation method

For the installation of Tengine, it seems that the official only provide source package, only compile and install.

For nginx installation, can be installed online through the online software source, you can also compile the source code to install

On-line installation generally uses a common CPU instruction set to compile Nginx, this method does not make good use of the current CPU instruction set, and configuration files, executables can not customize the installation directory. It is not convenient for mass deployment at scale.

Source compilation can be compiled by specifying the parameters of GCC with the instruction set of the CPU, which can be used to improve performance with the instruction set of the CPU, and the installation directory can be customized, and the rpath can be packaged into a company-specific RPM package by modifying the executable file, which is suitable for batch deployment.

For GCC available parameters, refer to the official GCC manual

optimization of basic configuration

#自动设置Nginx启动的worker数量, the default is the processor number of CPUs (not the core number)
Worker_processes Auto;
#自动将Nginx的worker进程绑定到CPU的processor上, 1.9.10 supports auto,tegine default support
Worker_cpu_affinity Auto;
#设置所有worker的open files number, if not set by default to the size of the system ulimit-n
Worker_rlimit_nofile 100000;
#开启pcre JIT function, need to turn on JIT function when compiling pcre
Pcre_jit on;
events{
Use Epoll;
Worker_connections 8192; The maximum number of connections per worker, this number is not only the connection between Nginx and client, but also the number of connections between Nginx and backend server, which should be noted when configuring Worker_rlimit_nofile>worker_ Connections*workers
Accept_mutex off; Suggest closing the Accept_mutex mechanism on a website with a large number of visits
}

Client_header_buffer_size 8k;
Client_header_timeout 10;
Client_body_buffer_size 256k;
Client_body_timeout 10;
Large_client_header_buffers 4 8k;
Client_max_body_size 20m;
Send_timeout 10;
Keepalive_timeout 30;
Keepalive_requests 5000;
Reset_timedout_connection on;
Log_format Access

the various buffer in the configuration file

Proxy_pass

Proxy_connect_timeout 60;
Proxy_send_timeout 60;
Proxy_read_timeout 60;
Proxy_buffer_size 64k;
Proxy_buffers 4 64k;
Proxy_busy_buffers_size 128k;
Proxy_temp_file_write_size 512k;
Proxy_next_upstream Error timeout Invalid_header http_503 http_404 http_502 http_504;
Proxy_max_temp_file_size 32m;
Proxy_intercept_errors on;

Fastcgi_pass

Fastcgi_connect_timeout 30;
Fastcgi_send_timeout 15;
Fastcgi_read_timeout 15;
Fastcgi_buffer_size 64k;
Fastcgi_buffers 4 64k;
Fastcgi_busy_buffers_size 128k;
Fastcgi_temp_file_write_size 512k;

estimation of maximum connection number for single Tengine

Rational use of Proxy_cache and Fastcgi_cache

So_reuseport && TCP Fast Open

About the principles of So_reuseport and TCP Fast open and what they do, you can find them on your own Google

This reuseport function Tengine 2.1.0 support, nginx until 1.9.1 support Reuseport

The So_reuseport feature was originally supported by more than kernel 3.9, but Red Hat supported the feature at the beginning of its 6.5 release, so the use of the CentOS 6 series of children's shoes would only be possible if the upgrade kernel was larger than the 6.5 version.

The TCP Fast Open feature is not supported in Rhel 6, Really need more than 3.7 of the kernel to support, there is need to use this feature of children's shoes can upgrade the kernel, upgrade can install the Uek kernel, or through Elrepo install a high version of the kernel, notify the need to recompile tengine or nginx to support TFO, the other Haproxy 1.5 has supported TFO features

reasonable use of TMPFS

In the process of using nginx, always consider the use of proxy_cache,fastcgi_cache, but also produce a variety of temporary files, in most cases, the operations engineer will use physical disk to undertake this part of the work, but this will generate additional IO, The important thing is that the response speed is lower than the memory, so in the case of various caches, as well as various temp, the following suggestions can be consulted:

A) cache server with Nginx
Under the precondition of using 1U DELL R630 and Memory 128G, if the cached data is less than 64G, consider using TMPFS directly as the Proxy_cache or Fastcgi_cache storage medium. PCIE-SSD can also be used to cache storage media when the number of caches is very large
b) Place Nginx-generated temporary files into TMPFS
I use the procedure to write the Mount Tmpfs command into the Nginx startup script. Readers can do their own research, the following for self-initiated script
Note TMPFS If you use echo {} >/proc/sys/vm/drop_caches, you cannot release the memory that TMPFS consumes unless you empty the Tmpfs

#! /bin/sh# description: startup script for tengine# chkconfig: 2345  55 25path=/sbin:/bin:/usr/sbin:/usr/bindesc= "Tengine daemon" name=tenginedaemon=/opt/websuite/ tengine/sbin/nginxconfigfile=/opt/config/tengine/nginx.confpidfile=/opt/run/tengine/$NAME. pidSCRIPTNAME=/etc/ init.d/rc.tenginetestpath=/opt/websuitecachedir=/opt/websuite/tengine/cachetempdir=/opt/websuite/tengine/ tempset -e[ -x  "$DAEMON"  ] | |  exit 0do_start ()  {  $DAEMON  -c  $CONFIGFILE  | |  echo -n  "Tengine already running"}do_stop ()  {  $DAEMON  -c $ configfile -s stop | |  echo -n  "Tengine not running"}do_reload ()  {  $DAEMON  -c $ configfile -s reload | |  echo -n  "Tengine can ' T reload"}do_mount_ramdisk ()  { mount -t  tmpfs tmpfs  $TEMPDIR/CLIENT -O&NBsp;defaults,size=32m,uid=websuite,mode=755 mount -t tmpfs tmpfs  $TEMPDIR/proxy - o defaults,size=32m,uid=websuite,mode=755 mount -t tmpfs tmpfs  $TEMPDIR/fastcgi  -o defaults,size=32M,uid=websuite,mode=755 mount -t tmpfs tmpfs  $TEMPDIR/ hmux -o defaults,size=32m,uid=websuite,mode=755 mount -t tmpfs tmpfs $ Cachedir/proxy -o defaults,size=512m,uid=websuite,mode=755 mount -t tmpfs tmpfs   $CACHEDIR/fastcgi -o defaults,size=512m,uid=websuite,mode=755}do_umount_ramdisk ()  {  umount  $TEMPDIR/client umount  $TEMPDIR/proxy umount  $TEMPDIR/fastcgi   umount  $TEMPDIR/hmux umount  $CACHEDIR/proxy umount  $CACHEDIR/fastcgi}case  "$"  in start)  echo -n  "starting  $DESC:  $NAME"  if [ $ (mount|grep   $TESTPATH |wc -L)  -eq 0 ];then        do_mount_ramdisk fi do _start echo  "."  ;;  stop)  echo -n  "stopping  $DESC:  $NAME"  do_stop if [ $ (mount| grep  $TESTPATH |wc -l)  -gt 0 ];then        do_ umount_ramdisk fi echo  "."  ;;  reload)  echo -n  "reloading  $DESC  configuration ..."  do_reload echo   "."  ;;  restart)  echo -n  "restarting  $DESC:  $NAME"  do_stop sleep 1  do_start echo  "."  ;;  *)  echo  "usage:  $SCRIPTNAME  {start|stop|reload|restart}"  >&2 exit  3 ;; Esacexit 0

5, Nginx common deployment Scenarios

static resource Web site

TENGINE+PHP-FPM

Tengine+tomcat

tengine building resin clusters with Hmux

Tengine Building a Web cluster with KVM

Haproxy/lvs+tengine with the use of

6. Monitoring

tengine Basic Status Output

output more data with Nginx-module-vts

7. Other

Distributed Cache

Relieve robot Pressure

Dynamic Update upstream

The role played by Tengine in the varnish architecture

This article is from the "Hongtashan" blog, make sure to keep this source http://wenzengliu.blog.51cto.com/9378751/1828689

Tengine/nginx performance optimization and its talk