Remote Server monitoring and restart Solution

Abstract: If your server's main services are killed from time to time, how can we ensure normal service as much as possible?

A problem occurred on a server a few days ago. The main processes (including web services, top, and VI commands) were occasionally killed. After analysis, the system of the server encountered a problem, must be reinstalled. This server is mounted with a very important web service. To ensure the normal operation of the service, you must migrate the service.

However, it still takes some time for the new server to be in place. How can we ensure the normality of the service during this time? Manual monitoring is time-consuming and labor-intensive, and it is impossible to monitor data from. It is feasible to use crontab, but the problem is that the minimum running interval of crontab is 1 minute, and the service has a high chance of failure within 1 minute. the maximum service downtime is 1 minute, great impact. It is feasible to use the script execution method, but it will also be killed, which cannot be used for continuous monitoring. Is there any good way?

The solution is to monitor the service status on other servers (this is easy to implement) and remotely execute scripts to restart the service.

1. Monitor Service Status

There are multiple ways to monitor service status, such as remote retrieval of system process list analysis and HTTP requests. The simplest way is to determine the service availability through the HTTP Request status.

1) First, write a simple PHP page on the monitored server (replaced by Server 1 below), and only output fixed content.

<? Php <br/> echo "alive"; <br/>?>
2) access the Web page with a Perl script on the monitoring server (Server 2 is used later). If the obtained content is inconsistent with the determined content, the service is considered unavailable (you can also use the HTTP status code to determine whether the service is unavailable ). The following code is check_down.pl:

Use lwp: simple; <br/> Use posix qw (strftime); </P> <p> sub convert_format_time {<br/> my $ time = shift; <br/> my $ UTC = strftime ("% Z", localtime); <br/> $ UTC = substr ($ UTC, 0,-2 ); <br/> return strftime ("% Y-% m-% d % H: % m: % s", localtime (8-$ UTC) * 60*60 + $ time); <br/>}</P> <p> while (true) {<br/> open ($ FH, "> check_result.out") or die "$! "; <Br/> $ result = lwp: simple: Get (" http://xxx.xxx.xxx.xxx/check_down.php "); <br/> $ time = convert_format_time (time ); </P> <p> if ($ result ne 'alive') {# server error <br/> Print $ FH "SERVERDOWN! -". $ Result. "-". $ time. "\ n"; <br/>}< br/> else {# Server Alive <br/> Print $ FH "serveralive. -". $ time. "\ n"; <br/>}</P> <p> close ($ FH); <br/> sleep (1); <br/>}
2. remotely execute the restart script

If the service is detected to be unavailable, the following is the most critical step: remotely execute the restart script.

1) The following is the shell script check_down.sh that is detected and restarted on Server 1.

#! /Bin/bash <br/> php_procs = 'psaux | grep PHP-FPM | grep-V grep | WC-l' </P> <p> If [$ php_procs-lt 1] <br/> then <br/>/etc/init. d/nginx start <br/>/etc/init. d/php5-fpm start <br/> fi
2) We need to call this script on Server 1 on Server 2. We can use SSH, but the key is that you do not need to enter a password when executing the SSH command.

You do not need a password to execute the SSH command as follows:

• Run "ssh-keygen-t rsa" on Server 2 and press Enter.
• Go to the/root/. Ssh directory on Server 1, Run "viauthorized_keys", and append the content in the/root/. Ssh/id_rsa.pub file on Server 2 to this file.
• You can directly log on to server 2 using the "ssh root @ IP address of Server 1" method, to remotely execute the restart script, Run "ssh root @ IP address of Server 1 '/path/check_down.sh '".

3) Adjust the Perl script and add the call to remotely execute the restart script when the server is down.

If ($ result ne 'alive') {# server error <br/> $ restart_result = 'ssh Root \ @ ip'/usr/local/bin/check_down.sh ''; <br/> Print $ FH "SERVERDOWN! -". $ Result."-". $ time." \ n "; <br/> Print $ FH" $ restart_result "." \ n "; <br/>}
Success!

Start the Perl script "Perl check_down.pl &" on Server 2, and then you can see the result in the output log file:

Server Alive.-21:05:25 <br/> server down! -- 21:05:26 <br/> starting nginx: nginx. <br/> * Starting PhP5 FPM... <br/>... done. </P> <p> Server Alive. -21:05:29

Summary:

The above solution is simple, but shell, Perl, and PHP scripts are compiled, which provide different functions. It is also important to consider how to solve the problem and how to bypass the problem without any possibility.