Monitoring clusters with SNMP

MonitoringThe
Performance of hosts in your cluster is somewhat a boring task. I 've
Seen various programs written to access system usage informations
Handmade methods, such as filtering output of commands, or reading
Fields from files in the '/proc' directory. These days I have to write
A program to retrieve informations, such as the CPU load, disk and
Memory usage, from a bunch of hosts, and I 've found that SNMP was
Good candidate to accomplish this task.

The Simple Network Management Protocol (SNMP)
Is an application layer protocol that facilitates the exchange
Management information between network devices. It is part of
Transmission Control Protocol/Internet Protocol (TCP/IP) Protocol
Suite. SNMP enables network administrators to manage network
Performance, find and solve network problems, and plan for Network
Growth.

I'll introduce what I 've found about SNMP in the following sections,
Including the installation, configuration of the agent, and how to use
The manager the pull informations from the agent.

1. Installation

On my Ubuntu box I just issued one line of command to install all the stuffs:

$ Sudo aptitude install SNMP snmpd

And on other platforms this shoshould be easy, too.

2. Configuration

Edit the configuration file of the SNMP agent:

$ Sudo VI/etc/snmp/snmpd. conf

Change the first step of access control section from:

# Sec. Name source community
Com2sec paranoid default public
# Com2sec readonly default public
# Com2sec readwrite default private

To:

# Sec. Name source community
Com2sec paranoid localhost public
Com2sec readonly 10.0.0.0/16 public
# Com2sec readwrite default private

This allows hosts belonging to the local network 10.0.0.0/16 to access this box's information in readonly mode.

Then restart the service:

$ Sudo/etc/init. d/snmpd restart

And you can test it:

$ Snmpwalk-V 2C-C public localhost System
SNMPv2-MIB: sysdescr.0 = string: Linux linkup 2.6.22-14-generic #1 SMP sun Oct 14 23:05:12 GMT 2007 i686
SNMPv2-MIB: sysobjectid.0 = OID: NET-SNMP-MIB: netsnmpagentoids.10
DISMAN-EVENT-MIB: sysuptimeinstance = timeticks: (1098742) 3:03:07. 42
SNMPv2-MIB: syscontact.0 = string: Root <root @ localhost> (Configure/etc/snmp/snmpd. Local. conf)
...

However, we 've not finished yet. If you issue the following command,
Which seems to be identical to the former, you'll be surprised to see
The result (substitute 10.0.12.4 with your IP address ):

$ Snmpwalk-V 2C-C public 10.0.12.4 System
Timeout: no response from 10.0.12.4

This is due to we didn't specify an IP address to listen to.
Default, snmpd listen to the loopback interface. To change this, append
The following line to the configuration file:

Agentaddress 10.0.12.4: 161

And restart the service, the problem will be solved.

3. Read informations

After set up on all your nodes, you may want to gather
Informations from them. The 'snmp 'package contains a bunch of tools
For sending SNMP requests to the agent, and handling the responses.
For example:

'Snmpport' retrieve a subtree of management values using SNMP getnext
Requests. This is useful for observing and debuging your scripts.

'Snmpget' communicates with a network entity using snmp get requests. This is used to retrieve the informations you need.

All the commands need a OID argument. You can search 'snmp MIB 'with your favorite search engine to learn about it.

To retrieve the CPU load, memory and disk usage from hosts, you need
Find the corresponding mibs for them. I find that my Linux box don't
Give result on the CPU load MIB (. 1.3.6.1.2.1.25.3.3.1.2 ).
So I have to get it with the ucdavis MIB. Another thing needs to be
Noticed is that from the hrstoragetable (. 1.3.6.1.2.1.25.2.3) You can
Get both the memory and disk usage. issue the following command and
Check which index is corresponding to your physical and virtual memory:

$ Snmpwalk-V 2C-C public 10.0.12.4. 1.3.6.1.2.1.25.2.3.1.3
HOST-RESOURCES-MIB: hrstoragedescr.1 = string: memory buffers
HOST-RESOURCES-MIB: hrstoragedescr.2 = string: real memory
HOST-RESOURCES-MIB: hrstoragedescr.3 = string: swap space
HOST-RESOURCES-MIB: hrstoragedescr.4 = string :/
HOST-RESOURCES-MIB: hrstoragedescr.5 = string:/sys
HOST-RESOURCES-MIB: hrstoragedescr.6 = string:/boot
HOST-RESOURCES-MIB: hrstoragedescr.7 = string:/home
HOST-RESOURCES-MIB: hrstoragedescr.8 = string:/Media/sda1
HOST-RESOURCES-MIB: hrstoragedescr.9 = string:/Media/sda5
HOST-RESOURCES-MIB: hrstoragedescr.10 = string:/Media/sda6
HOST-RESOURCES-MIB: hrstoragedescr.11 = string:/sys/kernel/security

The result indicates that the index of 2 is for my physical memory, and 3 for my swap space.

Following is a example to retrieve the CPU load, memory and disk usage from hosts:

#! /Bin/sh

Work_dir = $ (dirname $0)

Cpu_load_mib =. 1.3.6.1.2.1.25.3.3.1.2
Storage_unit_mib =. 1.3.6.1.2.1.25.2.3.1.4
Storage_size_mib =. 1.3.6.1.2.1.25.2.3.1.5
Storage_used_mib =. 1.3.6.1.2.1.25.2.3.1.6
Cpu_load_ucd_mib =. 1.3.6.1.4.1.2021.10.1.5.1

For host; do
. $ Work_dir/$ host

Echo "CPU usage"
Cpu_load =
If [$ platform = 'win32 ']; then
Cpu_count =
For I in $ cpu_indexes; do
Load = $ (snmpget-V $ snmp_version-C $ snmp_community $ host/
$ Cpu_load_mib. $ I | cut-D ''-F 4)
Cpu_count = $ ($ cpu_count + 1 ))
Cpu_load =$ ($ cpu_load + $ load ))
Done
Cpu_load = $ ($ cpu_load/$ cpu_count ))
Else
Cpu_load =$ (snmpget-V $ snmp_version-C $ snmp_community $ host/
$ Cpu_load_ucd_mib | cut-D ''-F 4)
Fi
Echo $ cpu_load

Echo "memory usage"
Mem_size =
Mem_used =
For I in $ memory_indexes; do
Unit = $ (snmpget-V $ snmp_version-C $ snmp_community $ host/
$ Storage_unit_mib. $ I | cut-D ''-F 4)
Size = $ (snmpget-V $ snmp_version-C $ snmp_community $ host/
$ Storage_size_mib. $ I | cut-D ''-F 4)
Used = $ (snmpget-V $ snmp_version-C $ snmp_community $ host/
$ Storage_used_mib. $ I | cut-D ''-F 4)
Mem_size = $ ($ mem_size + $ unit/1024 * $ size ))
Mem_used = $ ($ mem_used + $ unit/1024 * $ used ))
Done
Echo $ mem_size $ mem_used

Echo "disk usage"
Disk_size =
Disk_used =
For I in $ disk_indexes; do
Unit = $ (snmpget-V $ snmp_version-C $ snmp_community $ host/
$ Storage_unit_mib. $ I | cut-D ''-F 4)
Size = $ (snmpget-V $ snmp_version-C $ snmp_community $ host/
$ Storage_size_mib. $ I | cut-D ''-F 4)
Used = $ (snmpget-V $ snmp_version-C $ snmp_community $ host/
$ Storage_used_mib. $ I | cut-D ''-F 4)
Disk_size = $ ($ disk_size + $ unit/1024 * $ size ))
Disk_used = $ ($ disk_used + $ unit/1024 * $ used ))
Done
Echo $ disk_size $ disk_used

Done

Following is two sample configuration files to the former script:
$ Cat 10.0.15.141
Platform = Win32
Snmp_version = 2C
Snmp_community = Public

Cpu_indexes = '2 3 4 5'
Memory_indexes = '8'
Disk_indexes = '2 3 4'

$ Cat 10.0.15.99
Platform = Linux
Snmp_version = 2C
Snmp_community = Public

Cpu_indexes =''
Memory_indexes = '2'
Disk_indexes = '4'

4. Other Tools

Tkmib is useful to inspect the MIB tree. on Ubuntu you can install it from aptitute.

MRTG monitor SNMP network devices and draw pretty pictures showing how much traffic has passed through each interface.

Cacti is regarded as the best graph front-end for SNMP Network and the substitution to mrtg.