Supervisor Management process Superlance to process status alarm

Supervisor Introduction

First, introduce supervisor. Supervisor (http://supervisord.org/) is a client/server service developed with Python and is a process management tool under the Linux/unix system that does not support Windows systems. It is easy to listen, start, stop, restart one or more processes. With supervisor management process, when a process is accidentally killed, Supervisort hear the process after death, will automatically pull it back, it is convenient to do the process of automatic recovery function, no longer need to write their own shell script to control

The reason for writing this article is a requirement for the first two days ... Hey, there's no way to transport the dog ....

Environment: centos7.4

Installing Supervisor

Pip Install Supervisor

Because it is a library developed by Python, it is convenient to install it directly with PIP.
Supervisor when the installation is complete, three execution programs are generated: SUPERVISORTD, Supervisorctl, echo_supervisord_conf, respectively, the daemon service for the supervisor (for receiving process management commands), The client (for communicating with the daemon, sending instructions for the management process), and generating the initial profile program.

Configure Supervisor

Create a directory, initialize the configuration file

mkdir/etc/supervisorecho_supervisord_conf >/etc/supervisor/supervisord.conf

Master configuration file Parameters

[Unix_http_server]file=/tmp/supervisor.sock; UNIX socket file, SUPERVISORCTL will be used; chmod=0700; Socket file mode, the default is 0700;chown=nobody:nogroup; The owner of the socket file, format: Uid:gid; [Inet_http_server]; HTTP server, provide Web management interface; port=127.0.0.1:9001; Web Management background Run IP and port, if open to public network, need to pay attention to security; username=user; Login Admin background user name; password=123; Login admin background password [Supervisord]logfile=/tmp/supervisord.log; Log file, default is $CWD/SUPERVISORD.LOGLOGFILE_MAXBYTES=50MB; Log file size, exceeding the rotate, default 50mblogfile_backups=10; Log files retain the number of backups by default 10loglevel=info; Log level, default info, others: debug,warn,tracepidfile=/tmp/supervisord.pid; PID file Nodaemon=false; Whether to start in the foreground, the default is false, that is, to start the minfds=1024 in daemon way; The minimum value of the file descriptor that can be opened, default 1024minprocs=200; The minimum number of processes that can be opened, default 200; The below section must remain with the config file for RPC; (Supervisorctl/web interface) to work, additional interfaces; Added by defining them in separate rpcinTerface:sections[rpcinterface:supervisor]supervisor.rpcinterface_factory = Supervisor.rpcinterface:make_main_ Rpcinterface [Supervisorctl]serverurl=unix:///tmp/supervisor.sock; Through the UNIX socket connection Supervisord, the path is consistent with the file of the unix_http_server part; serverurl=http://127.0.0.1:9001; Connect the Supervisord via HTTP; Contains additional configuration files [include]files = Relative/directory/*.ini; It could be *.conf or *.ini.

　　

Managing a Process

All managed process profiles are placed in the same directory, and the master configuration file is included in it.

Mkdir/etc/supervisor/config.dvim/etc/supervisor/supervisord.conf    [include]    files =/etc/supervisor/ Config.d/*.conf

　

Create an Nginx configuration file below

' daemon off; '  3true true==/etc/supervisor/logs/ supervisord-=/etc/supervisor/logs/supervisord-nginx-error.log

To start supervisor.

At first there was a pit, command =/usr/local/nginx/sbin/nginx I wrote it like this, and it started. Nginx because this is the background to run the Nginx, however supervisor can not manage the background running process, so the Nginx Daemon off guard off, let him block in the foreground bash run, convenient supervisor to manage.

 [[email protected]/]# supervisorctl statusnginx RUNNING PID  17541 , uptime 0 : 03 : 42   [[email  Protected] /]# netstat-ntlpactive Internet connections (only servers) Proto recv
    -q send-q Local Address Foreign address State pid/program name TCP  0  0  0.0 . 0.0 : 80  0 .0 . 0.0 :* LISTEN 17541 /nginx:maste R 

At this time the Nginx PID is consistent.

Common SUPERVISORCTL Commands

supervisorctl Status     <name>  <name>  Restart supervisorctl Reload restart     Supervisord main process 

Web interface Management

Turn on Web Access

vim/etc/supervisor/supervisord.conf  [inet_http_server]          Port=0.0.  0.0:9001         username=user              password=123   

All right, that's it. Supervisor management process, the following began to say the alarm, some time, the process inexplicably quit, and then immediately be supervisor to pull up, causing some problems arise, want to know immediately that the process has been restarted how to do? That's when you can use Superlance.

Superlance Introduction

Superlance is a set of command-line tools based on the Supervisor event mechanism, which implements many practical process monitoring and management features not implemented by supervisor itself, including memory monitoring, HTTP interface monitoring, email and SMS notification mechanisms. Similarly, the superlance itself is written using Python

Superlance command

Superlance is a collection of command-line tools that includes the following commands:

• Httpok
  By timing a GET request to an HTTP interface, determine whether a process is in a normal state based on the success of the request, or restart the process if it is not normal.
• Crashmail
  Sends a message alert when a process exits unexpectedly.
• Memmon
  Sends an email alert when the memory footprint of a process exceeds the set threshold.
• Crashmailbatch
  Similar to Crashmail alarms, but messages will be synthesized and sent over time to avoid mail bombing.
• Fatalmailbatch
  When a process does not start multiple times successfully, it enters the fatal state, at which point the message alert is sent. As with Crashmailbatch, synthetic alarms are performed.
• Crashsms
  When a process quits unexpectedly send SMS alarm, this text message is also sent through the email gateway

 1 . When Supervisord is started, if our listener is configured as autostart= true, listener will be started as a child of supervisor.  2 . After the listener is started, it writes a stdout ready   "  3  .listener is in the ready state, When the event generated by Supervisord is in the acceptable events of listener's configuration, Supervisord will send the event to that listener.  4  .listener after receiving the event, we can, based on the head of the event, Body inside the data, do a series of processing. We operate according to the contents of the event, judging, extracting, alerting and so on.  5 . After all the work was done, listener needed to write a message to his stdout  resultnok   ", Supervisord after receiving this message. Just know that the listener processing event is complete. 

Event supported by Supervisord

Process_state process state changes process_state_starting process state transitions from other states to being started (Supervisord has startsecs configuration items in the configuration item,   This means that the program needs to run at least a stable x seconds at startup to assume that the program is running normally, and that the program state is starting in this X-second process_state_running process state is being started by transitioning to running Process_state_backoff   The process state is transitioning from being started to a failed process_state_stopping process state that is transitioning from running to being stopped process_state_exited process state is transitioning from running to exiting process_state_stopped The process state is converted from being stopped to already stopped (the difference between exited and stopped is that exited is the program itself exits, and stopped is controlled for its exit) Process_state_fatal process state is converted from running to failure Process_ State_unknown unknown process state remote_communication communicates with Supervisord using Supervisord RPC interface Process_log process produces log output, Includes standard output and standard error output process_log_stdout process produces standard output Process_log_stderr process produces standard error output process_communication process log output contains and Process_ The standard output of the Communication_stdout process contains and the standard error output of the Process_communication_stderr process contains and supervisor_state_change_running Supervisord start supervisor_state_change_stopping Supervisord stop tick_5 every 5 seconds trigger tick_60 every 60 seconds trigger tick_3600 every 3600 trigger process  _group Supervisord Process Group changed process_group_added new Supervisord process Group process_group_removed deleted Supervisord process Group

　　

Installing Superlance

Also Python program, direct PIP installation

Pip Install Superlance

Now that you have the event feature above, configure an email alert below, and send an email notification immediately after the Nginx is inexplicably restarted.

Here is to say, CentOS under how to use the command line to send mail directly, I have never used the default comes with, also not good use. Recommend a more useful.

SendEmail, written in Perl language is also packaged, only need to provide the sender's mailbox account password SMTP service recipients can be.

Configure it first

wget http://caspian.dotconf.net/menu/software/sendemail/sendemail-v1.56.tar.gztar zxvf Sendemail-v1.  About . TAR.GZCP sendEmail-v1. /sendemail/usr/bin/

OK, e-mail test, here I have to hide the mailbox information .....

" SendEmail " " haha " -xu Sender Mailbox-  XP Sender Mailbox Password -F -sender-t recipient -s Sender SMTP server -u topic -< C12>m content -Xu Sender user name -

I use the 163 QQ mail inbox, Test no problem.

OK now all you have to do is configure a supervisor configuration file to listen to the status changes of the process.

[Email protected]/]# cat/etc/supervisor/config.d/mail.conf [eventlistener:crashmail-exited]command=crashmail-a-S "/usr/bin/sendemail-f [email protected]-t [email protected]-s smtp.163.com-u ' nginx '-xu [email protected]-XP xxxxxxx X-m "-m [email Protected]events=process_state_exitedredirect_stderr=false

　　

After adding the good one process configuration file, Supervisorctl reload reboot

It's already two processes in running.

Let's test kill the Nginx process.

PS aux |  -91765917660  

And take a look at supervisor.

At this time the Nginx PID has changed, indicating that after the kill was pulled up again.

Also soon received the mail alarm. Hey..

Write this article, in addition to combat operations also have an hour ... I forgot to eat at noon ...

Well, it is here, the whole process I was the actual combat, this method is absolutely feasible, as long as I do to configure, then no problem.

A few days ago got a VPS and got a blog ... Hey, after the blog park their own blog are synchronized up. Gentlemen, don't squirt me. Traffic:) Address: http://www.binglansky.com

Supervisor Management process Superlance to process status alarm