Filter nodes inaccessible to Hadoop using Shell scripts
The hp1 cluster recently used, because the maintenance staff of the cluster is not powerful, the node will always drop one or two after a while. Today, we found that HDFS is in protection mode when Hadoop is restarted.
I decided to filter out all the inaccessible nodes in the slaves node, so I wrote a small script and recorded it here for convenience in the future.
PS: written in C Shell
The Code is as follows:
#! /Bin/csh
If ($ # argv <1) then
Echo "Usage: $0 host_file"
Exit 1
Endif
Set NODES = 'cat $1'
Foreach NODE ($ NODES)
Ping-q-c 1 $ NODE>/dev/null
If ($? = 0) then
Echo $ NODE
Endif
End
By the way, a script for ssh check is included:
#! /Bin/csh
Set nodes = 'cat slafs'
Foreach I ($ nodes)
Echo $ I
Ssh $ I exit
End
Build a Hadoop environment on Ubuntu 13.04
Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1
Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)
Configuration of Hadoop environment in Ubuntu
Detailed tutorial on creating a Hadoop environment for standalone Edition
Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)