Default Configuration and performance of nginx and php-fpm-TCPsocket or unixdomainsock

A few days ago, I saw a blog post saying that the server where php is located frequently creates TCP short connections in the case of high concurrency, And the 2MSL time of the server where php is located is too long, resulting in no port available, the system cannot create TCPsocket, but a large number of errors are reported. The solution provided by the blogger is to reduce the 2MSL time, clear the TCP connection in TIME_WAIT status as soon as possible, and clear the port.

A few days ago, I saw a blog post saying that the server where php is located frequently creates TCP short connections in the case of high concurrency, And the 2MSL time of the server where php is located is too long, resulting in no port available, the system cannot create TCP socket, but a large number of errors are reported. The solution provided by the blogger is to reduce the 2MSL time, clear the TCP connection in TIME_WAIT status as soon as possible, and clear the port.

A few days ago, I saw a blog post saying that the server where php is located frequently creates TCP short connections in the case of high concurrency, And the 2MSL time of the server where php is located is too long, resulting in no port available, the system cannot create TCP socket, but a large number of errors are reported. The solution provided by the blogger is to reduce the 2MSL time, clear the TCP connection in TIME_WAIT status as soon as possible, and clear the port. At the same time, I wrote the reason why persistent connections are not needed at the end of the article. But is this really the best solution? Is there any other way to do better?

Similar experiences
The reason why I am so interested in this article is that some time ago,Last CourseAfter a similar optimization process, we briefly describe our server architecture.

A rough webgame Architecture

I am sure you are familiar with this architecture diagram. It is also relatively simple. The "frontend" nginx anti-proxy is responsible for communicating with the player's http request. This is a persistent connection. Short connections are used to communicate with backend game regions. This means that each http request must be re-established with the backend nginx) the request is closed after the backend nginx is processed. The backend nginx and php are on the same server. The communication configuration is as follows:

// NGINX reversed upstream server1 {server 10.10.10.1 max_fails = 2 fail_timeout = 30 s; # app1 server 10.10.10.2 max_fails = 2 fail_timeout = 30 s; # app2} // backend NGINX // Nginx default configuration http://trac.nginx.org/nginx/browser/nginx/trunk/conf/nginx.conflocation ~ \. Php $ {root html; fastcgi_pass 127.0.0.1: 9000; fastcgi_index index. php; fastcgi_param SCRIPT_FILENAME/scripts $ fastcgi_script_name; include fastcgi_params;} // PHP-FPM // The PHP-FPM defaults to setting The address on which to accept FastCGI requests .; valid syntaxes are:; 'IP. add. re. ss: port '-to listen on a TCP socket to a specific address on a specific port; 'Port'-to listen on a TCP socket to all addresses on a specific port ;; '/path/to/unix/socket'-to listen on a unix socket .; note: This value is mandatory. listen = 127.0.0.1: 9000

In this architecture diagram, the reverse generation assumes the starting and ending roles. The front is the user, the back is the web app server, the persistent connection user, and the short connection backend. If the number of users is 1 W, and the number of users is 1 W, port 80 is used on the anti-proxy server. However, for every requests, the reverse proxy must re-use the server port and the backend nginx 80 to create a tcp socket to process the requests. If the 2MSL time is more than one minute, and the interval between the two requests is very short, the two requests are about 5 seconds (or even 2-3 seconds) to the server where we want to play the game ). In this case, the anti-proxy port should be 1 W in 5 seconds, 2 W in 10 seconds, 3 W in 15 seconds, 4 W in 20 seconds, 5 W in 25 seconds, and 6 W in 30 seconds !!!, The previously used port has not been recycled. After receiving many user requests by the nginx reverse proxy, you cannot create a tcp connection with backend nginx because there is no port resource... Take one of the game regions as an example:

Regarding this question, I have consulted other Daniel on Sina Weibo. The tengine Director Tao Shu Du's predecessors and the great frog's predecessors @ 120 told me that nginx will support ngx_http_upstream_keepalive in version 1.1.4, around March 1, it was version 1.1.16 dev. Our O & M team applied 1.1.16 to servers in some communities for a few days. The effect was good, the TCP connection between Nginx and port 80 on the target server is in the ESTABLISHED status, and the TIME_WAIT is almost no and stable. We almost plan to use dev for this performance improvement. Then nginx1.2Stable version is released, so this version is used. As for the number of TCP connections ESTABLISHED between nginx and the backend app server on reverse generation, that is, the number of ports (the status of ESTABLISHED is maintained at full load) nginx worker_processes * (number of keepalive parameters of upstream1 server *) + (number of keepalive parameters of upstream2 server *) + ...), Not many.

As shown in the preceding figure, the number of ports occupied is small. Is there anything else? First look at the socket creation, listening, data receiving and sending, closing the socket connection with the client flowchart:

For example, nginx reverse generation and backend nginx are the same as php-fpm and mysqld socket models.
Nginx anti-generation is a client relative to php-fpm. In this way, each of their new socket requests will follow the process on the left, over and over again .... Each socket creation, port and other resource application, three-way handshake with the server, four-way handshake when the server is closed, port recycling ....
After switching to persistent connections, for example:

Only the sub-processes of php-fpm (or nginx anti-generation) and mysqld create one TCP connection respectively, and then send and recv data, port occupation is also a few (each fpm sub-process will maintain a TCP persistent connection with mysqld ).

Similarly, the communication between nginx and php-fpm on the backend server is the same, but the two requests are not as large as the reverse generation. The difference is that the IP address is the loopback address 127.0.0.1.
Since it is a loop address, both services run on the same machine. Since it is the same machine, why not use the socket-unix domain socket for inter-process communication?
What is socket? Excerpt a description:

A Socket can be defined as the endpoints of two application communication channels. A Socket endpoint can be described using a Socket address. The Socket address structure consists of an IP address, a port, and a protocol (TCP or UDP ). The http protocol can be implemented through socket, and the socket is implemented on the transport layer. From this perspective, the socket is between the application layer and the transport layer. However, as a process communication mechanism, the operating system allocates a unique socket number, which depends on the communication protocol. However, this communication protocol is not only tcp or udp, but also other protocols.

What is the difference between tcp socket and unix domain socket on the same server?
For the two programs that communicate between processes, the unix domain socket process will not go to the TCP layer, and will communicate directly in the form of files and stream socket. If it is a TCP socket, you need to go to the IP layer.

For non-identical servers, more TCP sockets are used.

As for the differences between localhost \ 127.0.0.1 and network IP addresses, I accidentally found a blog with mysql as the verification to show that localhost does not go through the TCP/IP layer, which is different from 127.0.0.1. Finally, he thought he was wrong. His reasons are as follows:

(The following are all in linux, windows does not have unix domain socket)
When mysql is connected to the local machine, the-h parameter is not added:

When mysql connects to the local machine, add the-h parameter and the value is localhost:

When mysql connects to the local machine, add the-h parameter and the value is 127.0.0.1:

The classmate used the mysql tool and the result to understand the difference between the derivation of localhost and 127.0.0.1. This is a problem in the aspect. I believe that, this is the behavior of the mysql program. If the-h parameter is not added, or the value of the-h parameter is not in the IP format, and my. when specifying the mysql socket path in cnf, the unix domain socket is directly used to connect to the server. Of course, this is also my guess. If you haven't verified it, just listen and don't believe it.

In view of the above understanding, the server architecture configuration change is as follows:

// Configure upstream server1 {keepalive 10 single in NGinx reverse generation; // see nginx official wiki. Remember to check the E version, http://wiki.nginx.org/NginxHttpUpstreamModule server 10.10.8.97 max_fails = 2 fail_timeout = 30 s; # app1 server 10.10.8.99 max_fails = 2 fail_timeout = 30 s; # app2 server 10.10.8.85 max_fails = 2 seconds = 30 s; # app3} // NGINX configuration // rules for obtaining the PHP extension, applicable to location ~ ^ ([^.] + \. Php) ($ | /. *) {fastcgi_pass unix:/var/run/php5-fpm.sock; fastcgi_index index. php; include fastcgi_params;} // php-fpm configuration; Note: This value is mandatory. listen =/var/run/php5-fpm.sock // the path of fastcgi_pass in nginx must be the same, and the directory must have the corresponding read and write permissions.

So far, the optimization is complete. If mysql_pconnect is used for php-fpm and mysql, it is best to use static mode for the subprocess generation mode of php-fpm, if dynamic mode is used, the number of mysql connections may be full, which is also related to the connection timeout time of the mysql service. It is easy to avoid proper adjustment.
However, mysql_pconnect is not used yet. The main reason is that some transactions in our code fail to be processed after the transaction processing is started, and we forget to write the rollback statement. In the case of short connections, if the client does not submit the ROLLBACK or COMMIT command, mysql automatically rolls back the previous transaction. However, after persistent connections are used, different requests use the same MYSQL connection handle. When each transaction is enabled, automatic MYSQL commit is disabled, that is, the set autocommit = 0 statement, this statement will commit the previous transaction. This is a big problem when we forget to write and roll back the code and directly return the results. It is also the only reason why we have not used MYSQL_pconnect currently. (I plan to find the code that does not write the rollback statement in the near future, fix it, and continue to use mysql_pconnect)

In fact, php-fpm uses APC to cache php files and variable data, which are also optimized (if there is time, it will be continued ).

Looking back, I can't understand the solution provided by the student at the beginning of the article. I still can't understand the shortcomings of persistent connections from the reasons he gave, even if it solves the problem of TIME_WAIT, however, each time you create a TCP socket, three handshakes are performed when you connect to the server, and four handshakes when you close the TCP socket are overhead. Of course, shortening the 2MSL time is also a good way to make better use of server resources.

Finally, we adjusted the optimized server architecture as follows:

It seems a little different from the title of this article. In fact, I 'd like to say that I cannot understand why the default configurations provided by nginx and php-fpm are all TCP socket communication, why not default settings for unix domain socket? If we want to make it easier for a different server, the default IP address is also the loopback address.
In addition, in nginx's default configuration, the matching rules for the PHP file in the uri request are still old. Previously, security problems caused by NGINX and PHP configuration occurred, although it is not an error of nginx, nginx can also provide more rigorous examples, but it still does not.

Fortunately, in UBUNTU 12.4, nginx's default configuration has been greatly improved, whether it is matching the uri rules or the interaction between nginx and php-fpm:

# Location ~ \. Php $ {# fastcgi_split_path_info ^ (. + \. php )(/. +) $; // Like 1 # NOTE: You shoshould have "cgi. fix_pathinfo = 0; "in php. ini ### With php5-cgi alone: # fastcgi_pass 127.0.0.1: 9000; // like 3 # With php5-fpm: # fastcgi_pass unix:/var/run/php5-fpm.sock; // like 3 # fastcgi_index index. php; # include fastcgi_params ;#}

PS: the last course is just a web programmer. This is just a Summary of the last course of study and the above knowledge. It is inevitable that there will be errors in understanding. Please ask haihan.

Original article address: nginx, php-fpm default configuration and performance-TCP socket or unix domain socke, thanks to the original author for sharing.