monitoring hadoop cluster with nagios

recommend that you install Cloudera Manager on a Hadoop cluster, which provides real-time statistics on CPU, hard disk, and network load. (Cloudera Manager is a component of Cloudera Standard Edition and Enterprise Edition. The Enterprise Edition also supports rolling upgrade.) After Cloudera Manager is installed, the Hadoop administrator can run MapReduce tasks

report hardware components and corresponding failure rate status graph:Can obviously see the hard disk failure rate of the highest, reached 84%, so for operation and maintenance, if you can count the failure cases in peacetime work, and write them as an automated repair script, it will have great significance.If you can see a little further, you can think: can you make a hardware fault detection and repair system? (requires the cooperation of hardware manufacturers), I only do this here, if you

First of all, to ask, what is CDH?To install a Hadoop cluster that deploys 100 or even 1000 servers, package I including hive,hbase,flume ... Components, a day to build the complete, there is to consider the system after the update asked questions, then need to CDH Advantages of the CDH version:Clear Version DivisionFaster version updateSupport for Kerberos security authenticationDocument Clarity (Official

8649239.2. 11.71 }Modified to:/**/239.2. 11.71 8649239.2. 11.71 }2. Configure gmetad.confVim/etc/ganglia/gmetad.confData_source "My cluster" localhostModified to:Data_source "My Cluster" 192.168.10.128:86493. Restart Service required:/etc/init.d/ganglia-Monitor Restart/etc/init.d/Gmetad restart/etc/init.d/apache2 restartIf you encounter a situation where apache2 cannot be restartedVim/etc/apache2/apache2

java_home=/path/to/java command in console2 storing the data to be retrieved in HDFs$ bin/hadoop fs-put Urldir UrldirNote: The first Urldir is a local folder, a URL data file is stored, one URL per lineThe second urldir is a storage path for HDFs3 Starting the Nutch commandExecute the following command under the Nutch_hone/runtime/deploy directory$ bin/nutch Crawl Urldir–dir crawl-depth 3–topn 10After the command executes successfully, the crawl dire

Contact Hadoop has been 1.5, during the accumulation of some hadoop operations experience, has always wanted to develop a Hadoop monitoring system, just recently the laboratory has a related project, took the opportunity to study a bit, here to summarize the Hadoop

configuration file are: Run the ": WQ" command to save and exit. Through the above configuration, we have completed the simplest pseudo-distributed configuration. Next, format the hadoop namenode: Enter "Y" to complete the formatting process: Start hadoop! Start hadoop as follows: Use the JPS command that comes with Java to query all daemon processes: Star

Speaking of Hadoop and HBase Cluster Monitoring, we all know and use third-party monitoring tools, such as cacti, ganglia, and zabbix. Zenoss is used for playing more deeply. These tools are indeed good and can play a major role, but after a long time, I always feel that the monito