Generate Awstats charts with Zabbix and automate operations with Python Zabbix API

Awstats as a log analysis software, the function is good, but the interface is too simple, there is no chart function, here I took a flexible method, the Awstats analysis results (PV, hits (number of files), bandwidth, Visits (standalone IP)) is added to Zabbix and a trend chart is generated through Zabbix.

In the first two articles, the use of our team awstats and its way of working are briefly described: after Awstats analyzes each site, it generates a "database" file in "awstats012016.txt" format The Awstats Display page is the data generated from the file.

1. Awstats Log analysis in multi-server multi-site situations

2. Slow improvement of dynamically generated pages in Awstats CGI mode

The idea of this article is to get the data we want from the ' database file ' in this text format, and then add it to the Zabbix through a custom script, eventually satisfying the need to generate a PV trend chart.

The key to accomplishing this task is to analyze the content format of the data file like ' Awstats052016.txt ' (PS: In the author's "Years" shell experience, "Analysis source file format" and "Generate destination file format" both "format" Takes up a significant portion of your daily shell programming. Pull away O (∩_∩) o~)

The first is the custom script as Zabbix key, from the corresponding ' data file ' to obtain the values of PV, hits, bandwidth, visits. Use the shell to implement the following

Cat web_pv.sh

#!/bin/sh# from the Awstats database file such as Api/awstats052016.txt to get yesterday's PV statistics (because Awstats itself is counted to yesterday's log) #by   ljk   20160506#blog    http://kaifly.blog.51cto.com/#shell脚本的 $1 $ 2 represents the site name (format such as www or BBS) and statistics (pv  files   bytes   Standalone IP) basedir= '/usr/local/awstats-7.4/result ' date_f1=$ (date  +%m%Y -d  ' 1 day ago ') date_f2=$ (date +%y%m%d -d  ' 1 day ago ') cd  $basedir/$1# There is a little trick in the usage of awk below, after matching the specified item, to stop searching for the rest of the content. This can save a lot of time for larger files content= ' awk  ' $1 ==  "' $date _f2 '"  {{print} {exit}} '  awstats$ Date_f1\.txt ' case $2 in     pages ')          echo  $content |awk  ' {print $2} ';;      #pv      "hits")         echo   $content |awk  ' {print $3} ';;      #hits/Number of documents      "bandwidth")         echo  $content |awk  ' {print $4} ';;      #bandwidth/byte      "visits")          echo  $content |awk  ' {print $5} ';;      #visits/Independent ip    *)          echo  "Unknow value";; Esac

Then add the user-defined key in the Zabbix ' userparameter.conf ' file of the server where the Awstats resides and restart the ZABBIX_AGENTD

Userparameter=web_pv[*],/bin/sh/usr/local/zabbix/etc/zabbix_agentd.conf.d/web_pv.sh $

Then try to get these values through the Zabbix_get command at the Zabbix_server end, the key format is "web_pv[site name, monitoring item", for example

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/7F/F9/wKioL1czRTXDNt-9AAA2gArdX7k085.png "title=" QQ picture 20160511223952.png "alt=" Wkiol1czrtxdnt-9aaa2gardx7k085.png "/>

The value can be taken to indicate that the custom key is OK.

The next step is to add the item for each site in Zabbix, which is implemented via Python (the ZBX interface passes the data through JSON, and processing JSON python is much more convenient than the shell)

Here you need to read the API documentation for Zabbix Https://www.zabbix.com/documentation/3.0/manual/api and view the ZABBIX database structure (make sure it's foolproof)

First create a template in the Zabbix named template site PV, this step is created manually

Then start adding items through Python automation

cat shells/add_web-pv_item_to_ zabbix.py

#!/bin/env python3 "" "adds/Updates the 4 (pages,hits,bandwidth,visits) item of each site to Zabbix   ' TEMPLATE SITE-PV ' by  ljk  20160507 "" "Import os,requestsbasedir= '/usr/local/services/awstats-7.4/result/' items= [' pages ', ' hits ', ' bandwidth ', ' visits ']url= ' http://192.168.1.199/api_jsonrpc.php ' zbx_api_headers={' Content-type ' : ' Application/json-rpc '}     #定义通过zabbix  api must be taken when the header# obtained token for ZABBIX API certification, Using the user name password request API Generation # Generation method please refer to the API documentation, there is a token, you can save the account password Authentication api_auth= "738024dfda33cc6020fb1f5e3617"   #这里我在前期实验的时候, You have added several item manually, so here is the item that already exists in the template so that you can filter out the itemexist_items_filter={     when you create it later #通过zabbix  api Query the already existing web_pv[*,*] item, here is the JSON format filter condition      "Jsonrpc":  "2.0",      "Method":  "Item.get",     "params": {          "Output":[             "name",         ],         "Search": {              "Key_": "WEB_PV"          }    },     "auth":api_auth,     "id":  0}exist_items=requests.post (Url,headers=zbx_api_headers,json=exist_items_filter) os.chdir (basedir) sites= Os.listdir (path= '. ') Def create_item ():    for site in sites:         for item in items:             if site+ '-' +item not in exist_items.text:                  #先给不同情况下的units和multiplier赋值                  if item== ' pages '  or item== ' hits ':                     units= ' million '                      multiplier=0.0001                 elif item== ' Bandwidth ':                     units= ' B '                       multiplier=1                 else:                     units= '                      multiplier=1                  #定义创建item的json数据格式                  num=10                 create_item_post={                      "Jsonrpc":  "2.0",                      "Method":  "Item.create",                       "params": {                          "name":  site+ ', ' +item,                          "Key_":  "Www_ pv["+site+ ', ' +item+"] ",                          "HostID":  "10134",                           "Type": 0,                          "Value_type": 3,                           "History": 7,                          "Delay": 28800,                          "units":  units,                          "Applications": [774],                          " InterfaceID ": " 0 ",                          "Formula": multiplier                     },                      " Auth ": api_auth,                      "id": num                 }                 try:                     create_item_result=requests.post (url,headers=zbx_api_headers,json=create_item_post)                                    #打印处理每个条目的结果                      print (' { }-{}: return_code {} details {} '. Format (site,item,create_item_result.status_code,create_ Item_result.json))                      nUm+=1                except:                      print (' {}-{}: error '. Format (site,item))                      import sys                     sys.exit (255) # Create_item () #update函数, in fact, I was in the execution of Create_item () when the name of the key was wrong, helpless in writing a update_item () Def update_item ():     num=100     #对应zbx the ID field in  api, optionally, to ensure that the value is different each time the API is called (this is done in a self-increment way)       #定义更新item的json数据格式     update={          "JSONRPC":  "2.0",         "method":  "Item.update",          "params": {             " Itemid ": " ",            " Key_ ": " "         },         "Auth":  api_ auth,         "id": num    }      #取得site all the wrong item under  PV template (key starts with www_py), HostID value is actually templateid  of TEMPLATE SITE PV template    wrong_items_filter={             "Jsonrpc":  "2.0",             "method":  " Item.get ",            " params ": {                 "Output": ["Key_", "HostID"],                  "Search":  {"HostID": "10134", "Key_ ":" WWW_PV "}            },              "Auth": api_auth,              "id": 0        }     wrong_items=requests.post (Url,headers=zbx_api_headers,json=wrong_items_filter). JSON () [' Result ']      #wrong_items为list     for wrong_item in wrong_items:         if wrong_item[' HostID '] !=  ' 10119 ':      #img2从template  SITE PV inherited   So here each item corresponds to two records corresponding to the TEMPLATE SITE PV HostID : 10134 and Img2 of hostid:10119, so do not need to modify the Img2              update[' params ' [' itemid ']=wrong_item[' itemid ']            update[' params '] [' Key_ ']=wrong_item[' Key_ '].replace (' www ', ' web ', 1)                          try:                 update_item_result=requests.post (url,headers=zbx_api_headers,json=update)                  print (' {} ---- details {} '. Format (wrong_item[' Key_ '],update_item_ Result.json ()))                  num+=1            except:                 print (' {}-{}: error '. Format (Site,item))                 import sys                 sys.exit (255) #update _item ()

Subsequent batch generation of the image and generation screen can be done through the ZBX API, which is no longer listed

OK, finally look at the Zabbix generated by the beautiful map bar

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M02/7F/FC/wKiom1czPtagrjV3AADmK8dZcW8754.png "title=" QQ picture 20160511221629.png "alt=" Wkiom1czptagrjv3aadmk8dzcw8754.png "/>

This article is from "Endeavor K" blog, please be sure to keep this source http://kaifly.blog.51cto.com/3209616/1772377

Generate Awstats charts with Zabbix and automate operations with Python Zabbix API