Nginx performance optimization

: This article mainly introduces Nginx performance optimization. if you are interested in the PHP Tutorial, you can refer to it.

As a very popular and mature Web Server and Reserve Proxy Server, Nginx has a large number of performance optimization tutorials on the Internet, but different business scenarios vary widely. what configurations are most suitable for you, A large number of tests, practices, and continuous optimization and improvement are required. Recently, when the number of user calls broke through the million mark, some problems were encountered. although it was not too complicated, it took a long time to solve the problem and accumulated a lot of experience.

We have encountered this problem for some time. some customers have reported that the call has timed out, but we are normal in terms of system monitoring. it will certainly not time out only for dozens of milliseconds, I doubt whether it is the cause of the network, but after several times, I feel that this problem may not be sporadic, and there should be a deep-seated cause.

Because our services are oriented to enterprise customers, although each customer may have a very large call volume, each enterprise customer has just a few public IP addresses, even if there are thousands of customers in the future, nginx can also easily support these concurrent connections. Therefore, we first optimized the Nginx persistent connection from the network5 secondsChange5 minutesTo adjust the number of connection requests from the default 100 to 1000.

keepalive_timeout  300;keepalive_requests 1000;

After adjustment, passnetstat -anpThe command shows that the number of new connection requests decreases, indicating that the persistent connection has been used. However, after a period of time, the customer still found that the call time-out occurred. from the Nginx log, we can see that the request time still exceeded 1 s, or even about 20 s, as shown below:

In addition, monitoring on Zabbix finds a phenomenon. when the number of connection writing or active connections increases suddenly, the request time times out accordingly:

Check the application logs and find that the execution time is not long:

The time of statistics in the application is only the time from the start of the service to the execution result. the execution time of the Tomcat container is not counted yet. the execution path of the external request is as follows:

client --> Nginx -->  Tomcat --> App

Is there a problem with the execution of the Tomcat container itself? call the Tomcat request log and find that the execution before and after this time point is normal:

From the request path analysis, there must be some problems between Nginx and Tomcat. When I was troubleshooting this problem, I suddenly found a large number of timeout times out for about 30 s. I also observed that Zabbixconnection writingVery high, as shown below:

At the same timeTIME_WAITThere are a lot of connections. from the symptoms and packet capture analysis results, it should be that some customers have not enabled persistent connections, and we have set upkeepalive_timeout5 minutes, resulting in a large number of used connection wait times out, there are nearly 2000 at that time, Edit/etc/sysctl.confFile, add the following two parameters to reuse the connection:

Net. ipv4.tcp _ tw_reuse = 1 # indicates Enabling reuse. Allow TIME-WAIT sockets to be re-used for a new TCP connection. the default value is 0, indicating that it is disabled. net. ipv4.tcp _ tw_recycle = 1 # indicates to enable quick recovery of TIME-WAIT sockets in TCP connections. the default value is 0, indicating to disable it.

It soon fell below 200 after taking effect, as we can see from Zabbix monitoring,connection writingAnd connection activity' are obviously decreased, but the problem is not completely solved, you have to find other reasons.

Nginxreqeust_timeIt refers to the time when the client receives the first byte, calls the backend upstream server to complete business logic processing, and writes all the returned results back to the client, therefore, if the time for calling the upstream server can be printed out, it is easier to narrow down the scope of the problem. Fortunately, Nginx has two parameters that can print the time and IP address of the backend server request, in nginx. modify the log format in the conf file as follows:

# $ Upstream_response_time response time of the backend application server # $ upstream_addr backend server IP address and port number log_format main '$ remote_addr-[$ time_local] "$ request" ''$ status $ body_bytes_sent'' "$ request_time "" $ upstream_response_time "" $ upstream_addr "" $ request_body "';

Then observe the log, and it is obvious that most of the particularly long calls come from the same machine:

Check this machine and find that although the Java process is still in progress, the application has actually crashed and there are no real requests in. remove the application from the load balance and the problem is immediately mitigated:

This machine has actually crashed, but why does Nginx not recognize it? Further research shows that when Nginx calls upstream server, the timeout time is 60 s by default. our applications have very high response time requirements, and more than 1 s is meaningless. modify the default timeout value in the conf file. if the timeout value exceeds 1 second, the system returns the result:

# time out settings  proxy_connect_timeout 1s;  proxy_send_timeout   1s;  proxy_read_timeout   1s;

After running for a period of time, the problem has been basically solved, but you will still find request_timeFor more than 1 s and 5 S upstream_response_timeNone of them have timed out, indicating that the above parameters have taken effect.

Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.

The above introduces the Nginx performance optimization, including some content, hope to be helpful to friends who are interested in the PHP Tutorial.