Mongodb+python enables automatic collection of MongoDB slow query logs (with code)

1 Introduction

This is used in the production environment of a MongoDB slow query log automatic collection of scripts, the original idea to write this script is to facilitate the collection of slow query logs and facilitate analysis. Because the company's mognodb is not many, 4 units, so this small script is only applicable to ordinary production environment.

The main use of the page bootstrap for the front-end display, DataTable for analysis when used.

2 process

The main processes are:

1 connecting to MongoDB via Pymongo

2 Querying all slow query logs under System.profile using the Find () command

3 Use the JINJIA2 module to integrate the log information from the query into an HTML template.

4 The integrated HTML file, send the message to me as an attachment (using the SendMail module)

5 Write the script to crontab and send the message regularly every day.

Note: If you want to turn on the slow query log, open the method see: http://yucanghai.blog.51cto.com/5260262/1705195

3 Code

3.1 File structure

config.py # configuration file mongodb_slowlog.html #生成的html文件mongodb_slowlog. py # main file myhtml.py # HTML module

3.2 Code Information

1 config.py

#!/usr/bin/env python#-*-coding:utf-8-*-mail_to_list = [' [email protected] '] #收件人列表mail_host = ' smtp.163.com ' Mail_use r = ' xxx ' #发件人用户mail_pass = ' xxx ' mail_postfix = ' 163.com ' #mongodb user and password usename = ' root ' password = ' xxx ' # DB for the database to be analyzed ip_d B_list = [{' IP ': ' 10.18.96.86 ', ' Port ': 27017, ' db ': ' Test ', ' hostname ': ' test1 ', ' property ': ' Primary ', ' slow_log ': []},]

2 mongodb_slowlog.py

#!/usr/bin/env python# -*- coding: utf-8 -*-import pymongoimport bsonfrom  jinja2 import Templatefrom myhtml import html_strimport configimport  smtplibfrom email.mime.text import mimetextfrom email.mime.multipart import  Mimemultipartimport timeimport datetimedef _conn (Ip,port):    try:         conn = pymongo. Mongoclient ('%s '%ip,port)     except Exception,e:         conn = None        print e     return conn# converts the system time to the time  mongodb can recognize Def convert_time (t):     tm  = bson. Timestamp (t,1)     return tm.as_datetime () #发送邮件  --  with Accessories def send_mail ():     msg = mimemultipart ()     att = mimetext (Open (' mongodb_slowlog.html ', ' RB '). Read (), ' base64 ', ' gb2312 ')     att[' Content-type '] =  ' Application/octet-stream '     att[ "Content-disposition"] =  ' attachment; filename= "mongodb-slow-log%s.html" '%time.strftime ('%Y-%m- %d %h:%m:%s ')     msg.attach (att)     me =  ' MongoDB ' + ' < ' +config.mail_user+ ' @ "+config.mail_postfix+" > "    msg[' Subject '] =  ' mongodb slow log -- %s '%time.strftime ('%y-%m-%d %h:%m:%s ')     msg [' From '] = me    msg[' to '] =  '; Join (config.mail_to_list)     try:        server  = smtplib. SMTP ()         server.connect (config.mail_host)          server.login (cOnfig.mail_user,config.mail_pass)         server.sendmail (Me, Config.mail_to_list,msg.as_string ())         server.quit ()          return True    except Exception,e:         print e        return  false# Get slow log information Def profile_info (today,yesterday):    for ip_db in  Config.ip_db_list:        conn = _conn (ip_db[' IP '],ip_db[' Port '])          #用户验证         # auth_db = conn.admin         #auth_db. Authenticate ( Config.usename,config.password)         db_conn = getattr ( conn,ip_db[' DB ']) &NBSP;&NBSP;&NBSP;&NBSP;&NBsp;   if  db_conn.profiling_level ():              #获取     exclusions  getmore  and  comadn    slow log    in a day           ip_db[' Slow_log '].append (list (db_ Conn.system.profile.find ({' ts ': {"$lt": Today, "$gt":yesterday},  "Op": {"$nin": [' getmore ', ' Command ']}})))          #print   ip_db[' Slow_log ']          #解析获取到的日志到  html in         template =  template (HTML_STR)         rest_html = template.render ( db_info = ip_db)          #print   rest_html          #将获取到的html   Write as file, send to   user as Attachment          with open (' Mongodb_sloWlog.html ', ' W ')  as f:            f.write (rest_html)         print   ' send success! '  if send_mail ()  else  ' send failed! ' if __name__ ==  ' __main__ ':     today_time =  convert_time ( Datetime.datetime.today ())     yesterday_time = convert_time (today_time +  datetime.timedelta (Days=-1))     profile_info (Today_time,yesterday_time)

Note:

Can change according to their own needs, the idea is unchanged

3 myhtml.py

#!/usr/bin/env python# -*- coding: utf-8 -*-html_str = ' <! Doctype html>

4 mongodb_slowlog.html
Primarily for templates generated through myhtml.py

This script is written in a hurry, there is not perfect place, there is any doubt everyone can come to ask me

This article is from "a struggling small operation" blog, please be sure to keep this source http://yucanghai.blog.51cto.com/5260262/1705559
Mongodb+python enables automatic collection of MongoDB slow query logs (with code)