Use Python to implement a simple project monitoring and python project monitoring

Use Python to implement a simple project monitoring and python project monitoring

An interface system made in the company is mainly used to connect to third-party system interfaces. Therefore, this system will interact with projects of many other companies. The following is a very painful problem. The stability of the interfaces of so many companies varies greatly. When there is a large amount of traffic, some interfaces that do not work very well may encounter various errors.

This interface system has just been developed, and the entire system is at a relatively edge. Unlike other projects, there are logstores and text message alerts. If something goes wrong, users often report back, so my idea is to pick up python and write a monitor for this project. If a large number of errors occur during the call of a third-party interface, it indicates that the interface is faulty, and you can take faster measures.

The project also has a logstore. All info and error logs are scanned and written into the database every minute. The logstore uses mysql, and the table contains several important fields:

• Level Log level
• Message Log content
• File_name Java code file
• Log_time log time

If you have a logstore, you don't have to scan the log analysis in the online environment and start with the logstore. Because the logstore is scanned every one minute when it is online, I will go to the logstore and scan every two minutes. If a certain number of error logs are scanned, an alarm will be triggered, if there are only one or two errors, you can ignore them. That is, if a large number of error logs are generated in a short time, you can determine that the system is faulty. You can send an email in the alert mode. Therefore, you need to do the following:
1. Operate MySql.
2. send an email.
3. scheduled task.
4. logs.
5. Run the script.

After clarifying the above things, you can do it.
Database Operations

Using the MySQLdb driver, you can directly perform database operations, mainly query operations.
Obtain the database connection:
 

def get_con(): host = "127.0.0.1" port = 3306 logsdb = "logsdb" user = "root" password = "never tell you" con = MySQLdb.connect(host=host, user=user, passwd=password, db=logsdb, port=port, charset="utf8") return con

Obtain data from the logstore to obtain the data of 2 minutes before the current time. First, calculate the time based on the current time. Previously, there was a computing problem, and now it has been modified.
 

def calculate_time():  now = time.mktime(datetime.now().timetuple())-60*2 result = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(now)) return result

Then, query the data in the logstore based on the time and log level.
 

def get_data(): select_time = calculate_time() logger.info("select time:"+select_time) sql = "select file_name,message from logsdb.app_logs_record " \   "where log_time >"+"'"+select_time+"'" \   "and level="+"'ERROR'" \   "order by log_time desc" conn = get_con()  cursor = conn.cursor() cursor.execute(sql) results = cursor.fetchall()  cursor.close() conn.close()  return results

Send email

It is easy to send emails using python. You can use the standard library smtplib.
You can use another email address or enterprise email address to send emails by using the 163 email address. However, you must set the host and port address correctly.
 

def send_email(content): sender = "sender_monitor@163.com"receiver = ["rec01@163.com", "rec02@163.com"]host = 'smtp.163.com'port = 465msg = MIMEText(content)msg['From'] = "sender_monitor@163.com"msg['To'] = "rec01@163.com,rec02@163.com"msg['Subject'] = "system error warning" try:smtp = smtplib.SMTP_SSL(host, port)smtp.login(sender, '123456')smtp.sendmail(sender, receiver, msg.as_string())logger.info("send email success")except Exception, e:logger.error(e)

Scheduled task

A separate thread is used to scan logs every two minutes. If the number of logs at the ERROR level exceeds 5, an email is sent.
 

def task():while True:logger.info("monitor running") results = get_data()if results is not None and len(results) > 5:content = "recharge error："logger.info("a lot of error,so send mail")for r in results:content += r[1]+'\n'send_email(content)sleep(2*60)

Logs

Configure log. py for this small script so that logs can be output to files and the console.

# coding=utf-8import logging logger = logging.getLogger('mylogger')logger.setLevel(logging.DEBUG) fh = logging.FileHandler('monitor.log')fh.setLevel(logging.INFO) ch = logging.StreamHandler()ch.setLevel(logging.INFO) formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')fh.setFormatter(formatter)ch.setFormatter(formatter) logger.addHandler(fh)logger.addHandler(ch)

So, at last, this monitoring applet is like this app_monitor.py.

# coding=utf-8import threadingimport MySQLdbfrom datetime import datetimeimport timeimport smtplibfrom email.mime.text import MIMETextfrom log import logger  def get_con(): host = "127.0.0.1" port = 3306 logsdb = "logsdb" user = "root" password = "never tell you" con = MySQLdb.connect(host=host, user=user, passwd=password, db=logsdb, port=port, charset="utf8") return con  def calculate_time():  now = time.mktime(datetime.now().timetuple())-60*2 result = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(now)) return result  def get_data(): select_time = calculate_time() logger.info("select time:"+select_time) sql = "select file_name,message from logsdb.app_logs_record " \   "where log_time >"+"'"+select_time+"'" \   "and level="+"'ERROR'" \   "order by log_time desc" conn = get_con()  cursor = conn.cursor() cursor.execute(sql) results = cursor.fetchall()  cursor.close() conn.close()  return results  def send_email(content):  sender = "sender_monitor@163.com" receiver = ["rec01@163.com", "rec02@163.com"] host = 'smtp.163.com' port = 465 msg = MIMEText(content) msg['From'] = "sender_monitor@163.com" msg['To'] = "rec01@163.com,rec02@163.com" msg['Subject'] = "system error warning"  try:  smtp = smtplib.SMTP_SSL(host, port)  smtp.login(sender, '123456')  smtp.sendmail(sender, receiver, msg.as_string())  logger.info("send email success") except Exception, e:  logger.error(e)  def task(): while True:  logger.info("monitor running")  results = get_data()  if results is not None and len(results) > 5:   content = "recharge error："   logger.info("a lot of error,so send mail")   for r in results:    content += r[1]+'\n'   send_email(content)  time.sleep(2*60)  def run_monitor(): monitor = threading.Thread(target=task) monitor.start()  if __name__ == "__main__": run_monitor()

Run scripts

Run the script on the server and use the supervisor for management.
Install the supervisor on the server (centos6) and add the following configuration in/etc/supervisor. conf:

Copy codeThe Code is as follows: [program: app-monitor]
Command = python/root/monitor/app_monitor.py
Directory =/root/monitor
User = root

Then run supervisord on the terminal to start the supervisor.
Run supervisorctl on the terminal, go to shell, run status, and check the running status of the script.
Summary

This small monitoring idea is clear and can be modified, such as monitoring specific interfaces and sending SMS notifications.
Because there is a logstore, it is less difficult to scan logs in the online environment. Therefore, if there is no logstore, You need to scan the environment on your own. Be careful when you are in the formal online environment ~