Python Performance monitoring Graphite

First, Introduction

Graphite is a python-written web application, using the Django Framework, Graphite used to collect all the server's timely state, user request information, memcached hit rate, rabbitmq the status of the message server, The load state of the UNIX operating system, the graphite server needs to be updated approximately 4,800 times per minute, and the graphite is easy to use on any operating system using a simple text protocol and drawing capabilities.

There are three components of the graphite:

• Graphite-web:web interface

• Carbon: equivalent to network interface

• Whisper: Equivalent to RRDtool

Graphite Official documents:

Http://graphite.wikidot.com/documentation

http://graphite.readthedocs.org/en/latest/

Second, installation graphite

1. Install Epel source

RPM-IVH http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpmsed-i ' [email protected]^#@@ '/ Etc/yum.repos.d/epel.reposed-i ' [email protected]@#[email protected] '/etc/yum.repos.d/epel.repo

2, install the appropriate version of the Django package, the version is too high will appear bug

Yum Install Python-simplejsonwget https://kojipkgs.fedoraproject.org//packages/Django14/1.4.14/1.el6/noarch/ DJANGO14-1.4.14-1.EL6.NOARCH.RPMRPM-IVH django14-1.4.14-1.el6.noarch.rpm

3, installation Graphite

Yum Install Graphite-web Python-carbon python-whisper

4. Install MySQL Database

Yum install MySQL mysql-server mysql-pythonservice mysqld startchkconfig mysqld onmysqladmin-uroot Password 123456mysql- UROOT-P123456-E ' CREATE database graphite; '

5. Modify the Graphite configuration file

# cat >>/etc/graphite-web/local_settings.py << eofsecret_key = ' 123qwe ' allowed_hosts = [' * ']time_zone = ' as Ia/shanghai ' DEBUG = truedatabases = {' default ': {' NAME ': ' Graphite ', ' ENGINE ': ' DJANGO.DB.BACKENDS.MYSQ L ', ' USER ': ' Root ', ' PASSWORD ': ' 123456 ', ' HOST ': ' 127.0.0.1 ', ' PORT ': ' 3306 '}}from graphi Te.app_settings Import *eof

6. Synchronizing the database

Mkdir-p/opt/graphite/storagecd/etc/graphite-web/django-admin syncdb--settings=local_settings--pythonpath=. Yesroot[email protected]123456123456

7. Modify the Graphite Data Catalog

Chown-r Apache.apache/opt/graphite/storage

8. Start the service

/etc/init.d/carbon-cache startchkconfig carbon-cache on/etc/init.d/httpd startchkconfig httpd on

Third, visit the exhibition graphite

1, Chrome browser access to GHIPTE address:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/5C/11/wKiom1UaGbvgMgkrAAIEjT1B2Zk181.jpg "title=" 1.png " Width= "650" height= "322" border= "0" hspace= "0" vspace= "0" style= "width:650px;height:322px;" alt= " Wkiom1uagbvgmgkraaiejt1b2zk181.jpg "/>

2. Provide a script to monitor network card traffic

[[email protected] ~]# cat network_traffic.py #!/usr/bin/env pythonfrom  Subprocess import popen,pipeimport socketimport shleximport timeimport sysimport  osdef get_traffic (f):     p = popen (Shlex.split (f), stdout=PIPE,stderr= PIPE)     result = p.stdout.read ()     d = [i  For i in  result.split (' \ n ') [3:] if i]    dic_traffic =  {}    for i in d:         Devname = i.split (': ') [0].strip ()         receive =  i.split (': ') [1].split () [0].strip ()         Transmit =  I.split (': ') [1].split () [8].strip ()         dic_traffic[devname] =  {' in ': Receive, ' out ':transmit}    return dic_traffic if __name__ ==  ' __main__ ':     try:    HOST =  ' 127.0.0.1 ' port = 2003     s = socket.socket ()     s.connect ((host,port))      except:print  couldn ' t connect to % (server) s on port % (port) d,  Is carbon-agent.py running? "  % {' server ': HOST, ' Port ':P ort}sys.exit (1)     while True:     cur_traffic = get_traffic (' Cat /proc/net/dev ')     time.sleep (5)      five_s_traffic = get_traffic (' Cat /proc/net/dev ')     diff _dic = {}        for k in cur_traffic:     traffic_in = int (five_s_traffic[k][' in '])  - int (Cur_traFfic[k][' in '])     traffic_out = int (five_s_traffic[k][' out ')  - int ( cur_traffic[k][' out '])     diff_dic[k] = {"in": traffic_in, ' out ': Traffic_out}now  = int (Time.time ()) For k,v in diff_dic.items ():     net_in =   ' network.%s_in %s %s\n '  %  (k,v[' in '],now)     net_out =   ' network.%s_out %s %s\n '  %  (k,v[' out '],now)     s.sendall (net_in)     s.sendall (Net_out) time.sleep (5)

3, backstage mode running monitoring network card traffic script

[email protected] ~]# python network_traffic.py &

Iv. Installation of Diamond

Diamond: Collector, for data collection

Diamond's GitHub official site: Https://github.com/python-diamond/Diamond/wiki

1. Install Diamond

Yum install gcc gcc-c++ python-configobj python-pip pip install diamond==3.4.421 (sometimes not successful) if the download installation is unsuccessful you can use the following method wget https://pypi.python.org/packages/source/d/diamond/diamond-3.4.421.tar.gz#md5= 080ab9f52a154d81f16a4fd27d11093atar XF diamond-3.4.421.tar.gzcd diamond-3.4.421python setup.py Install

2. Configuration

CD/ETC/DIAMOND/CP diamond.conf.example diamond.conf Major modification of three profiles: [[email protected] diamond]# vim/etc/diamond/ Diamond.conf[[graphitehandler]]//59 Line host = localhost[[default]]//173 row interval = 10//time to collect [[email protected] Diamond ]# vim/etc/diamond/handlers/archivehandler.conf#log_file =./storage/archive.log//note this line [[email protected] diamond]# vim/etc/diamond/handlers/graphitehandler.conf host = localhost

3. Start the Diamond service

chmod +x/etc/init.d/diamond/etc/init.d/diamond Startchkconfig Diamond on

Five, continue to visit the display Diamond Automatic collection of information

1, Chrome browser access to GHIPTE address:

You will find that there is a servers directory in the graphite, this directory is the information that Diamond automatically collects

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/5C/0B/wKioL1UaG8fRUruvAAPm1blb_Gk327.jpg "title=" Diamond.png "width=" 650 "height=" "border=" 0 "hspace=" 0 "vspace=" 0 "style=" width:650px;height:320px; "alt=" Wkiol1uag8fruruvaapm1blb_gk327.jpg "/>2, where two Python scripts are available to collect httpcode of the Web site, is based on the diamond Way

[[email protected] httpcode]# ll total dosage 8-rwxr-xr-x 1 root root 1356 March 11:12 filerev.py-rwxr-xr-x 1 root root 3737 March 31 11:12 httpcode.py

3, run the collection Httpcode script

First delete the servers directory generated by the original diamond [[Email protected] httpcode]# rm -rf /var/lib/carbon/whisper /servers/then manually run the diamond's httpcode script [[Email protected] httpcode]# diamond -f -l -r  ./httpcode.py  -c /etc/diamond/diamond.conferror: pidfile exists. server  already running? #需要手动停止diamond服务 [[Email protected] httpcode]# /etc/init.d/diamond  stopStopping diamond:                                            [Determine][[email protected] httpcode]#  diamond -f -l -r ./httpcode.py  -c /etc/diamond/diamond.conf[ 2015-03-31 11:13:56,198] [mainthread] changed uid: 0  ()  GID: 0  () . [2015-03-31 11:13:56,198] [mainthread] loaded handler: diamond.handler.graphite.graphitehandler[2015-03-31  11:13:56,201] [MainThread] GraphiteHandler: Established connection to  graphite server localhost:2003. [2015-03-31 11:13:56,202] [mainthread] loaded handler:  diamond.handler.archive.archivehandler[2015-03-31 11:13:56,206] [mainthread] loading  collectors from: . [2015-03-31 11:13:56,209] [mainthread] loaded module: httpcode[2015-03-31  11:13:56,209] [mainthread] loaded collector: httpcode. httpcodecollector[2015-03-31 11:13:56,209] [mainthread] initialized collector:  Httpcodecollector[2015-03-31 11:13:56,210] [mainthread] skipped loading disabled  collector: httpcodecollector[2015-03-31 11:13:56,210] [mainthread] started task  scheduler. [2015-03-31 11:13: 57,211] [mainthread] stopping task scheduler. [2015-03-31 11:14:01,217] [mainthread] stopped task scheduler. [2015-03-31 11:14:01,217] [mainthread]  Exiting. If there is no error, the View browser will find a servers directory, but the directory is not generated, I am really puzzled. Originally this configuration was not started in the configuration file [[Email protected] httpcode]# vim httpcode.py......config = super (httpcodecollector, self). Get_default_config ()          Config.update ({             ' path ':       ' Weblog ',             ' enabled ':    ' True ' #开启此选项         }) if you collect with diamond, you do not need this option, Because diamond has a configuration file for the class, opening in the configuration file will look more uniform than opening in the script

4. Browser view

Chrome refreshes the GHIPTE Web page to see

Ghipte, servers, ec2-54-201-82-69, weblog (custom), HTTP, the following monitoring graphs appear

We can generate 200 status codes using Ab-c 100-n http://localhost/

Use the Refresh Ghipte browser page to generate a 304 status code

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/5C/0B/wKioL1UaHCijRVkoAAMpNTmsVaU913.jpg "title=" Manual httpcode.png "width=" 650 "height=" 322 "border=" 0 "hspace=" 0 "vspace=" 0 "style=" width:650px;height:322px; "alt=" Wkiol1uahcijrvkoaampntmsvau913.jpg "/>

• At present, the mainstream open source monitoring has cacti, Nagios, Zabbix and so on, the community is active, powerful

• Graphite Although functionally and in the community can not be compared with this, but in the flexibility is worth mentioning, the lightweight monitoring program, more importantly, graphite is written by Python, so in the troubleshooting, scripting, etc. will be very handy

• Also very grateful for more Python open source contributions!!!

This article is from the "Zheng" blog, make sure to keep this source http://467754239.blog.51cto.com/4878013/1626736

Python Performance monitoring Graphite