Ganglia configuration for monitoring system and Hadoop performance

First, Introduction

The Ganglia monitoring kit consists of three main sections: Gmond,gmetad, and web interface, often referred to as ganglia-web.

Gmond is a daemon that runs on every node that needs to be monitored, collects monitoring statistics, sends and receives statistics on the same multicast or unicast channel.

Gme Tad is also a daemon that periodically examines gmonds, pulls data from there, and stores their metrics in the RRD storage engine. It can query multiple clusters and aggregate metrics. RRD is also used to build the Web front end of the user interface.

Ganglia-web, as the name implies, should be installed on a machine with Gmetad running in order to read the RRD file.

In general, every machine on a cluster needs to install Gmond, as long as the machine responsible for displaying the Web interface installs the Gmetad.

Second, installation

1.apt-get Install Ganglia-monitor Ganglia-webfrontend

Pop-up dialog box Select Yes

You can see that the installation Ganglia-webfrontend includes Gmetad and RRDtool.

2. Linking to a Web server

When installing Ganglia-webfrontend, you have installed Apache Web server,ganglia-webfrontend The default installation location is/usr/share/ganglia-webfront, It is not possible to access the ganglia monitoring interface through a browser, you need to execute the following command to link ganglia files to Apache's default directory.

LN -s/usr/share/ganglia-webfront/var/www/ganglia

3. Copy the Ganglia webfrontend Apache configuration, using the following command to the correct location:

CP /etc/ganglia-webfrontend/apache.conf/etc/apache2/sites-enabled/ganglia.conf

Third, the configuration

1. Configure gmond.conf

Make the following changes:

/*If A cluster attribute is specified and then all Gmond the hosts are wrapped inside* of a tag. If you don't specify a cluster tag, then all will* is wrapped inside of a tag. */Cluster {name="Unspecified"owner="Unspecified"Latlong="Unspecified"URL="Unspecified"}

Modified to:

" My cluster " // set the cluster name yourself " Root "  "unspecified" "Unspecified" }

/**/239.2. 11.71  86491}

Modified to:

/**/239.2. 11.71 // comment out, multicast 192.168. 10.128 // Change to host IP 8649  1}

/**/239.2. 11.71  8649239.2. 11.71 }

Modified to:

/**/239.2. 11.71  8649239.2. 11.71 }

2. Configure gmetad.conf

Vim/etc/ganglia/gmetad.conf

Data_source "My cluster" localhost

Modified to:

Data_source "My Cluster" 192.168.10.128:8649

3. Restart Service required:

/etc/init.d/ganglia-Monitor Restart/etc/init.d/Gmetad restart/etc/init.d/apache2 restart

If you encounter a situation where apache2 cannot be restarted

Vim/etc/apache2/apache2.conf

Finally add a sentence ServerName localhost:80 can.

4. Now you can access Gnglia webinterface using the following command:

http://192.168.10.128/ganglia/

Iv. monitoring the performance of Hadoop clusters

The 1.Master node is done as described above and the IP address of all machines is added to the gmetad.conf. Slave nodes only need to install Ganglia-monitor and configure gmond.conf.

2. Configure the hadoop-metrics.properties on all machines

Modify the Hadoop configuration file/etc/hadoop/hadoop-metrics.properties, according to the instructions in the file, modify five places:

dfs.class=Org.apache.hadoop.metrics.ganglia.GangliaContext31dfs.period= -dfs.servers=192.168.10.128:8649Mapred.class=Org.apache.hadoop.metrics.ganglia.GangliaContext31mapred.period= -mapred.servers=192.168.10.128:8649Jvm.class=Org.apache.hadoop.metrics.ganglia.GangliaContext31jvm.period= -jvm.servers=192.168.10.128:8649Rpc.class=Org.apache.hadoop.metrics.ganglia.GangliaContext31rpc.period=Tenrpc.servers=192.68.10.128:8649Ugi.class=Org.apache.hadoop.metrics.ganglia.GangliaContext31ugi.period=Tenugi.servers=192.168.10.128:8649

3. Configure the hadoop-metrics2.properties on all machines

At the end add

*.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31*.sink.ganglia.period=Ten*.sink.ganglia.slope=jvm.metrics.gccount=zero,jvm.metrics.memheapusedm=both*.sink.ganglia.dmax=jvm.metrics.threadsblocked= -, jvm.metrics.memheapusedm= +namenode.sink.ganglia.servers=192.168.10.128:8649resourcemanager.sink.ganglia.servers=192.168.10.128:8649datanode.sink.ganglia.servers=192.168.10.128:8649nodemanager.sink.ganglia.servers=192.168.10.128:8649maptask.sink.ganglia.servers=192.168.10.128:8649reducetask.sink.ganglia.servers=192.168.10.128:8649

(The following six lines are configured according to their Hadoop situation)

Can also be configured on one machine and then uploaded to other machines.

4. After all configuration is complete, restart the Hadoop environment and restart the ganglia.

All machines need ganglia-monitor restart

Master also to do Gmetad restart and apache2 restart.

You can then see the Hadoop monitoring related content on the page.

Having done a wordcount program test, you can see that I ran the wordcount position with obvious network fluctuations.

In addition all installed after the bottom left corner of the page has a pie chart can not be displayed, installed the GD library or not

Install PHP5-GD

I hope you know how to tell me, thank you.

Ganglia configuration for monitoring system and Hadoop performance