Use Perl for system service monitoring and alarms

Generally, Web sites include many services and applications. We cannot know whether the system is running normally in real time, especially when the server goes down or the application goes down at night, will affect the business and
During user access, a set of system monitoring errors must be properly configured. At present, there are many software monitoring notifications and alarm services, and there are charges and free of charge, you can choose.

We will try to implement a service monitoring and alarm notification program by ourselves, so that we can use a very small price, and also make our services highly available and highly reliable.

[Monitoring principle]

Remote Service

For remote machines, we can have a monitoring server, or simply find a server that cannot be down as a monitoring server, so that we can monitor the services on other servers,
Remote monitoring is a much-needed method. Generally, remote monitoring is used to monitor whether servers and ports are open. For example, Apache of our Web Service is usually open to 80
Then, we can access port 80 of the server to determine whether Apache is working normally. If it cannot be connected, the service will be stopped.

Local Service

For the local machine, the monitoring process and log files are both feasible. Generally, services that work frequently for a long time, such as Apache will record the access information after each access
Access
In the access log file, if the file is not updated for a long time, you can suspect that the service has stopped (of course, it is not ruled out that no one has accessed the file during this period ). In addition, the local machine is
It is easy to see the process. For MySQL and other servers, the daemon process is always open.
Daemon, you can also confirm that the MySQL service has stopped.

　　Alarm notification

　　
When the service is stopped, the system maintenance personnel must be notified. In general, the SMS is the best via email or text message, but frequent text messages also make the maintenance personnel very depressed. This is called a text message bomb.
(Message bomb), so the mail may be a simple and practical method, and then Configure Outlook/Foxmail locally.
Regular reception and notification methods are quick, but after going home at night, you will not be able to receive emails, so the proper method is to send notifications by email during the day, the alarm Method for evening and weekend SMS notifications is more reasonable.

[Code implementation]

The specific code implementation can use a variety of code, C/C ++, Ruby, Python, PHP, as long as you can access files, socket
The code below can be constructed using Perl, because Perl is a good script language for system management, any Unix/Linux
The Perl engine is installed by default, which can be easily run on any machine. At the same time, Perl is flexible and has powerful CPAN
Package library, so it is very convenient to write code. It is also worth recommending in system management. Of course, it may be more convenient to use shell scripts for many system management tasks.

The following code implements remote monitoring, local Log File monitoring, and local process monitoring, but only the remote port monitoring method is used, because it can monitor multiple machines and services.
If you only need a single machine or want to monitor local processes, you can easily modify the main function. At the same time, the notification method mainly uses the mail Notification method, and the function implements the SMTP protocol for mail
Sending (because I found that Perl's built-in net: SMTP
It is not very reliable when performing type verification.) Of course, in terms of alarm notification, you can simply change it to sending text messages or calling SMS and email by time.

The Code mainly monitors services such as Apache, MySQL, memcache, and search (if you have one). You can add or delete different server monitors on this basis, you only need to add a constant to configure and modify the main function code.

Note: The following Perl code is successfully tested in the RHEL 4 + Perl v5.8.6 environment.

 

#! /Usr/bin/perl
Use IO: socket;
Use IO: file;
Use MIME: base64;

##############################
# Constant define (configure)
##############################
# Mail config
Use constant mail_addr => ('to' => 'webmaster @ example.com ', 'from' => 'webmaster @ example.com ');
Use constant smtp_info => ('host' => 'smtp .example.com ', 'user' => 'webmaster', 'Password' => 'pass ',
'Debug' => 1, 'bufsize' => 1024 );
# Common config
Use constant md5sum_file => '/tmp/_ monitor_md5sum_hash ';
Use constant apache_log_path => '/usr/local/apache2/logs/access ';
# Apache
Use constant apache_port => 80;
Use constant apache_servers => ('web1 .example.com ', 'web2 .example.com ');
# MySQL
Use constant mysql_port => 3306;
Use constant mysql_servers => ('db1 .example.com ', 'db2 .example.com ');
# Memcache
Use constant memcache_port => 11211;
Use constant memcache_servers => ('cache1 .example.com ', 'cache2 .example.com ');
# Search
Use constant search_port => 8000;
Use constant search_servers => ('search1 .example.com ');

##############################
# Server port is alive check
##############################
Sub check_server_alive {
My ($ server, $ port) = @_;

$ Sock = IO: Socket: iNet-> New (peeraddr => $ server, peerport => $ port, proto => 'tcp ', timeout => 3 );
If (! $ Sock ){
Return 0;
}
$ Sock-> close ();
Return 1;
}

##############################
# Check process is exist
##############################
Sub check_process_exist {
My $ proc_name = shift;
$ Line = '/bin/PS auxw |/bin/grep $ proc_name |/bin/grep-V grep |/usr/bin/WC-l ';
$ Line = ~ S/^ s + | S + $ // G;
If ($ line = 0 ){
Return 0;
}
Return 1;
}

##############################
# Check file MD5 fingerprint
##############################
Sub check_file_md5sum {
My $ Io, $ line;
$ Filename = shift;
@ Arr = Split (/S/, '/usr/bin/md5sum $ filename ');
$ Filehash = shift (@ ARR );
$ IO = IO: file-> New ();
$ Io-> open (md5sum_file, o_rdwr );
If (! ($ Line = $ io-> Getline ())){
$ Io-> syswrite ($ filehash );
$ Io-> close;
Return true;
}
If ($ line! = $ Filehash ){
$ Io-> truncate (0 );
$ Io-> syswrite ($ filehash );
$ Io-> close;
Return true;
}
Return true;
}

##############################
# SMTP Execute Command
##############################
Sub smtp_cmd {
My ($ sock, $ cmd, $ blocking) = @_;
My % smtpinfo = smtp_info;
My $ Buf, $ bufsize = $ smtpinfo {'bufsize'}, $ DEBUG = $ smtpinfo {'debug '};

$ Sock-> syswrite ($ cmd );
If ($ DEBUG = 1 ){
Print ">>>$ cmd ";
}
If ($ blocking = 1 ){
$ Sock-> sysread ($ Buf, $ bufsize );
If ($ Debug ){
Print "<$ Buf ";
}
}
}

##############################
# Send notice mail
##############################
Sub send_mail {
My ($ subject, $ content) = @_;
My $ sock;
My % mailaddr = mail_addr;
My % smtpinfo = smtp_info;
My $ DEBUG = $ smtpinfo {'debug '};

# Count date time
($ Sec, $ min, $ hour, $ day, $ Mon, $ year, $ wday, $ yday, $ isdst) = localtime (Time ());
　
$ Datetime = sprintf ("% s-% S % s: % s", "20". substr ($ year, 1, 2 ),
Length ($ Mon) = 1? "0 $ mon": $ Mon, length ($ day) = 1? "0 $ Day": $ day,
Length ($ hour) = 1? "0 $ Hour": $ hour, length ($ min) = 1? "0 $ min": $ min,
Length ($ Sec) = 1? "0 $ sec": $ Sec );
$ Subject. = "[$ datetime]";

# Connect to SMTP Server
$ Sock = IO: Socket: iNet-> New (peeraddr => $ smtpinfo {'host'}, peerport => 25, proto => 'tcp ', timeout => 10 );
$ Sock-> blocking (1 );

# Send SMTP command
If ($ DEBUG = 1 ){
Print "<". $ sock-> sysread ($ Buf, $ smtpinfo {'bufsize '});
}
Smtp_cmd ($ sock, "HELO locahost", 1 );
Smtp_cmd ($ sock, "auth login", 1 );
Smtp_cmd ($ sock, encode_base64 ($ smtpinfo {'user'}), 1 );
Smtp_cmd ($ sock, encode_base64 ($ smtpinfo {'Password'}), 1 );
Smtp_cmd ($ sock, "mail from: <". $ mailaddr {'from'}. ">", 1 );
Smtp_cmd ($ sock, "rcpt to: <". $ mailaddr {'to'}. ">", 1 );
Smtp_cmd ($ sock, "data", 1 );
Smtp_cmd ($ sock, "From:". $ smtpinfo {'from'}. "", 0 );
Smtp_cmd ($ sock, "to:". $ smtpinfo {'to'}. "", 0 );
Smtp_cmd ($ sock, "Subject: $ subject", 0 );
Smtp_cmd ($ sock, "$ content", 0 );
Smtp_cmd ($ sock, ".", 1 );
Smtp_cmd ($ sock, "quit", 0 );
$ Sock-> close ();

Return 1;
}

##############################
# Check Server Alive main
##############################
Sub monitor_main {
# Check Apache
Foreach $ item (apache_servers ){
If (! Check_server_alive ($ item, apache_port )){
Send_mail ("$ item Apache server is down", "$ item Apache server is down. Please timely restoration ");
}
}
# Check MySQL
Foreach $ item (mysql_servers ){
If (! Check_server_alive ($ item, mysql_port )){
Send_mail ("$ item MySQL server is down", "$ item MySQL server is down. Please timely restoration ");
}
}
# Check memcache
Foreach $ item (memcache_servers ){
If (! Check_server_alive ($ item, memcache_port )){
Send_mail ("$ item memcache server is down", "$ item memcache server is down. Please timely restoration ");
}
}
# Check search
Foreach $ item (search_servers ){
If (! Check_server_alive ($ item, search_port )){
Send_mail ("$ item search server is down", "$ item search server is down. Please timely restoration ");
}
}
}

##############################
# Main running
##############################

Monitor_main ();