Application of nginx lock mechanism in automatic server monitoring script

Introduction: This is a detailed page on the application of nginx lock mechanism in automatic server monitoring scripts. It introduces PHP, related knowledge, skills, experience, and some PHP source code.

Class = 'pingjiaf' frameborder = '0' src = 'HTTP: // biancheng.dnbc?info/pingjia.php? Id = 336635 'rolling = 'no'>

Http://my.nuaa.edu.cn/thread-120539-1-1.html

#! /Bin/sh
# Check PHP-FPM process and restart if down
# Create by zhengdongliang
# Path/data/SH/check_php_fpm.sh
# Crontab time */1 ****

If [-E/var/lock/subsys/502]
Then
Killall-9 curl 2>/dev/null
Killall-9 PHP-FPM 2>/dev/null
/Usr/local/webserver/PHP/sbin/PHP-FPM Start>/dev/null
Echo "['date + '% H % d % t''] PHP-FPM died with no response, all processes restarted">/tmp/log
Else
Touch/var/lock/subsys/502
If ['curl -- connect-Timeout 5-I http: // 192.168.93.129/2>/dev/null | grep '502 bad gateway'-C '! = '0']
Then
Killall-9 PHP-FPM 2>/dev/null
/Usr/local/webserver/PHP/sbin/PHP-FPM Start>/dev/null
Echo "['date + '% H % d % T'] PHP-FPM died with 502 Bad Gateway, all processes restarted">/tmp/log
Fi
Rm-F/var/lock/sub sys/502
Fi

Add as automatic execution:
# Crontab-e
*/1 *****/bin/sh/data/SH/check_php_fpm.sh>/dev/null 2> & 1

The system automatically executes check_php_fpm.sh every minute.

------------------------------------

Because the load capacity cannot meet the requirements, paper airplane servers have been migrated to nginx servers since last July, but PHP
After the eaccelerator extension is installed on MySQL 5.2, the server often encounters a 500 error and is upgraded to PhP.
5.3 series, although said 500 error is gone, but the new built-in PHP-FPM stability is not as good as playing FPM patch PHP-CGI, often occur all PHP-FPM process stuck not to accept
In any request, the front-end nginx keeps displaying the 502 Bad Gateway. It is very depressing to restore the normal status only after the PHP process is manually killed and restarted.
In
I visited countless forums online and switched to nginx English forum for a long time. I tried file descriptor restrictions, converted to APC, modified various FPM parameters, and other methods, but they were still ineffective.
Google wrote an article about how to use cron to schedule tasks and use curl to monitor the server for errors 502 every minute. If yes, the PHP process will be automatically restarted. Finally, you can only choose this method.
. After referring to the PHP script on the website, I changed it with Shell (If php fails, I will use PHP to kill PHP. Isn't it my own trouble? I decided to execute it in Shell language) and got the following code:
Next script:
#! /Bin/sh

If ['curl -- connect-Timeout 5-I http://my.nuaa.edu.cn/2>/dev/null | grep '502 bad gateway'-C '! = '0']
Then
Killall-9 PHP-FPM 2>/dev/null
Service PHP-FPM Start>/dev/null
Echo "['date + '% H % d % T'] PHP-FPM died with 502 Bad Gateway, all processes restarted">/path/to/log
Fi
Add a line to/etc/crontab:
* *** Root/path/to/script
The script is automatically executed at 0 seconds per minute.
Because the PHP process cannot receive any signal when it is stuck, even if the service PHP-FPM Stop command is used, it cannot be stopped. Therefore, you can only use the killall command to end the PHP process. In order to facilitate diagnosis and optimize stability in the future, the log record command is added to the script.
No
There is actually a time difference when the PHP process gets stuck to nginx and the 502 error is reported, because PHP and nginx are connected through TCP or socket. When PHP gets stuck, the request is not accepted.
Nginx does not receive similar information rejected by the request, but overlogs the request on the socket or TCP. In this case, the browser returns a "waiting for response" prompt, unless
The socket error is reported or the browser is closed, otherwise the wait will never stop, and the curl is the same. When the backlog reaches a certain value (Linux socket
The default value of backlog is 1024). in Linux, the system considers the socket to be invalid. If you try to send a request, the system will receive the resource temporarily
Unavailable. nginx will not report the 502 error until then. That is to say, it may take several minutes to report a 502 error when the website traffic is relatively small from the point where PHP is stuck to nginx.
Even longer, during this time, when the above script runs the curl command, curl will remain in the waiting state, and a new curl process will appear every minute, so that when the final socket
When the backlog value is reached, the group of curls will all get the 502 error return value, and kill all PHP processes and start the PHP service at the same time, that is, the PHP service is started
It moved many times. Because the number of worker processes started in the PHP-FPM configuration file is determined based on the actual memory situation, if the PHP service is started many times, it will directly lead to excessive memory usage
To avoid this situation, I added the following to the first line of the script:
Killall-9 curl 2>/dev/null
This avoids
Multiple PHP services are started, and several times more than normal PHP processes are running. However, in actual use, it is found that the time difference between the failure of PHP and nginx reporting error 502 may be required
It takes tens of minutes to half an hour, which is far beyond our tolerable range. in Linux, many services are created with their own names in the/var/lock/subsys/folder.
File to indicate whether you are running. I decided to add a lock mechanism to the monitoring script with the 502 error so that the script can identify the situation where PHP is stuck and nginx does not report the 502 error and
Restart the PHP process, so the monitoring script looks like this:
#! /Bin/sh

If [-E/var/lock/subsys/502]
Then
Killall-9 curl 2>/dev/null
Killall-9 PHP-FPM 2>/dev/null
Rm-F/var/lock/sub sys/502
Fi
Touch/var/lock/subsys/502
If ['curl -- connect-Timeout 5-I http://my.nuaa.edu.cn/2>/dev/null | grep '502 bad gateway'-C '! = '0']
Then
Killall-9 PHP-FPM 2>/dev/null
Service PHP-FPM Start>/dev/null
Echo "['date + '% H % d % T'] PHP-FPM died with 502 Bad Gateway, all processes restarted">/path/to/log
Fi
Rm-F/var/lock/sub sys/502
Foot
Before each curl operation, create a lock file "502" under/var/lock/subsys/to delete the lock file. When the lock file exists during each running, it means the previous
The script has not been run in minutes. That is to say, PHP is stuck, but nginx has not reported the 502 error. At this time, the curl and PHP-FPM processes are ended in sequence and the lock files are deleted, and subsequent commands
To restart the PHP service. Here, the ending sequence of the curl and PHP-FPM processes cannot be reversed because the previous script is still running. If the PHP-FPM process is terminated first, the curl will receive
502 error and restart PHP, And the PHP Startup operation after the current script may occur at the same time, resulting in the PHP service being started multiple times.
However, after careful consideration, we found that,
If the previous script runs too slowly after the curl is stopped, it will not run until the lock creation operation of the current script is complete until a row is deleted, after the current script creates the lock
The current script is actually running but the lock file does not exist. That is to say, the lock will no longer work, And the next script will re-create the lock and start curl probing.
Test the server status. If a 502 error occurs, the PHP service will be started multiple times. After repeated consideration, it is decided that if the lock of the previous script still exists during the current script running
The lock creation does not delete the lock, but ends the curl command so that the previous script continues to run to delete the lock, however, if the current script does not create a lock, it will restart the PHP service and record the error information before exiting.
The lock is created only when the lock of the previous script is deleted, so that the lock can correctly indicate the execution of the script. So I got the following script:
#! /Bin/sh

If [-E/var/lock/subsys/502]
Then
Killall-9 curl 2>/dev/null
Killall-9 PHP-FPM 2>/dev/null
Service PHP-FPM Start>/dev/null
Echo "['date + '% H % d % t''] PHP-FPM died with no response, all processes restarted">/path/to/log
Else
Touch/var/lock/subsys/502
If ['curl -- connect-Timeout 5-I http://my.nuaa.edu.cn/2>/dev/null | grep '502 bad gateway'-C '! = '0']
Then
Killall-9 PHP-FPM 2>/dev/null
Service PHP-FPM Start>/dev/null
Echo "['date + '% H % d % T'] PHP-FPM died with 502 Bad Gateway, all processes restarted">/path/to/log
Fi
Rm-F/var/lock/sub sys/502
Fi
In this way, it can be ensured that the server cannot work normally for less than one minute due to the freezing of the PHP process.
However, the stability of the PHP-FPM is always people feel pain, hope that the new server will be launched as soon as possible, there is enough performance to run Apache to obtain better stability.
If you have any of the above errors, you are welcome to make a picture ~

Love J2EE follow Java Michael Jackson video station JSON online tools

Http://biancheng.dnbcw.info/php/336635.html pageno: 10.