Ganglia monitoring the installation and deployment of Hadoop clusters [go]

Ganglia monitoring the installation deployment of Hadoop clusters First, installation environment Ubuntu Server 12.04 Installing the Gmetad machine: 192.168.52.105 Installing the Gmond Machine Device: 192.168.52.31,192.168.52.32,192.168.52.33,192.168.52.34,192.168.52.35,192.168.52.36,192.168.52.37,192.168.52.38,192.168 .52.105 Browse the machine for monitoring Web pages: 192.168.52.105 Second, Introduction The Ganglia monitoring kit consists of three main sections: Gmond,gmetad, and web interface, often referred to as ganglia-web. Gmond is a daemon that runs on every node that needs to be monitored, collects monitoring statistics, sends and receives statistics on the same multicast or unicast channel. Gme Tad is also a daemon that periodically examines gmonds, pulls data from there, and stores their metrics in the RRD storage engine. It can query multiple clusters and aggregate metrics. RRD is also used to build the Web front end of the user interface. Ganglia-web, as the name implies, should be installed on a machine with Gmetad running in order to read the RRD file. In general, every machine on a cluster needs to install Gmond, as long as the machine responsible for displaying the Web interface installs the Gmetad. Third, installation 1.apt-get mode installation First find a node as the Web display node, use the following command one-click installation: Apt-get-y Install Ganglia-monitor Ganglia-webfrontend Installing the Ganglia-webfrontend is equivalent to installing Gmetad, Ganglia-web, and RRDtool at the same time. The above installed version is 3.1.7, because Ubuntu official source only support to ganglia-webfrontend-3.1.7, from the use of the ganglia Web The Frontend version is also 3.1.7, the version is too low, the interface shown is very unfriendly, so later in the article I will describe how to manually install ganglia Web Frontend version 3.5.7 (currently the highest version has reached 3.6). If for convenience, for the monitoring of the interface requirements are not high, but also diagram easy, you can use the above command one-click installation. In addition to installing the Ganglia-webfrontend host, the other hosts only need to install Ganglia-monitor. 2. Linking to a Web server When installing Ganglia-webfrontend, you have installed Apache Web server,ganglia-webfrontend The default installation location is/usr/share/ganglia-webfront, It is not possible to access the ganglia monitoring interface through a browser, you need to execute the following command to link ganglia files to Apache's default directory. Ln-s/usr/share/ganglia-webfront/var/www/ganglia Iv. Configuration 1. Unicast and multicast before ganglia cluster configuration, it is first to understand unicast and multicast. Unicast: Can propagate across network segments, sending information only to designated machines. To configure as unicast you should specify one (or more) accepted hosts. Multicast: broadcasts are sent to all machines on the same network segment in the network segment where the machine is located. If you are using multicast transmission then you do not need to change anything because this is the default for ganglia package installation. The only thing to do is to point the Gmetad to one or several hosts running the GMO ND. It is not necessary to list each individual host, because the GMO ND is set to accept mode to include a list of all hosts and statistics for the entire cluster. For more detailed introduction and configuration of unicast and multicast, refer to: Ganglia Quick Start Wizard (translated from the official wiki) http://www.linuxidc.com/Linux/2013-11/92747.htm, or view the official wiki directly. 2. Cluster configuration My ganglia monitoring cluster uses unicast mode. All configurations are in the/etc/ganglia directory, and/ETC/GANGLIA/GMOND.CONF are configured on each node.The red font is the modified item.   Globals { Daemonize = yes Setuid = yes user = root/* Users running ganglia */ Debug_level = 0 Max_udp_msg_len = 1472 Mute = no deaf = No Host_dmax =/*secs */ Cleanup_threshold =/*secs */ Gexec = No Send_metadata_interval = 15/* time interval to send data */ } Cluster { Name = "HADOOP_HF"/* Cluster Name */ Owner = "Root"/* User running Ganglia */ Latlong = "Unspecified" url = "Unspecified" } Udp_send_channel { # mcast_join = 192.168.52.105/* Comment off multicast */ Host = 192.168.52.105/* Sent to the machine where the Gmetad is installed */ Port = 8649 TTL = 1 The boot ganglia uses the Apt-get method to install the ganglia, can be started directly by service mode. Service Ganglia-monitor start (each machine needs to start) Service Gmetad start (starts on the machine with Ganglia-webfrontend installed) Vi. Inspection Login Browser view: Http://192.168.52.105/ganglia, if hosts up is 9 that means the installation was successful. If the installation is unsuccessful, there are several useful debug commands: Start gmetad:gmetad-d 9 in debug mode To view the XML files that Gmetad collected: Telnet 192.168.52.105 8649 Vii. installation of Ganglia-web Because the 3.1.7 version of Ganglia-web was so ugly, I uninstalled the ganglia-webfrontend, and then I installed the Gmetad and installed the new version of ganglia-web-3.5.10. Official Installation Instructions:/HTTP Sourceforge.net/apps/trac/ganglia/wiki/ganglia-web-2#installation according to the official instructions, First extract the tar.gz file, then enter the extracted directory, modify the makefile file in the following four configurations: # location where Gweb should is installed to (excluding Conf, Dwoo dirs). Gdestdir =/var/www/ganglia# Gweb statedir (where conf dir and dwoo templates dir is stored) Gweb_statedir =/var/lib/gang lia# Gmetad RootDir (parent location of rrd folder) Gmetad_rootdir =/var/lib/gangliaapache_user = root (avoid having no space behind) Restart Gmetad. If the configuration has a problem after login Http://192.168.52.105/ganglia can be modified according to the error in the webpage. Generally, the file read and Write permission error, no RRD directory need to create their own, conf and the cache directory permissions need to be modified to 777. Monitoring interface: Eight, monitoring Hadoop cluster Modify the Hadoop configuration file/etc/hadoop/hadoop-metrics.properties, according to the instructions in the file, modify three places: dfs.class= Org.apache.hadoop.metrics.ganglia.GangliaContext31 Dfs.period=30 dfs.servers=192.168.52.105:8649 mapred.class=org.apache.hadoop.metrics.ganglia.gangliacontext31 Mapred.period=30 mapred.servers=192.168.52.105:8649 Jvm.class=org.apache.hadoop.metrics.ganglia.gangliacontext Jvm.class=org.apache.hadoop.metrics.ganglia.gangliacontext31 Jvm.period=30 jvm.servers=192.168.52.105:8649 all servers are modified to the machine IP installed as Gmetad. Restart Hadoop datanode:service hadoop-datanode restart restart Gmond:/usr/sbin/gmond restart

Ganglia monitoring the installation and deployment of Hadoop clusters [go]