Get the latest Datanode information from the cluster with SED and awk

since the remote Desktop is not so, the cluster appears missing block when the remote Desktop can not log on to see those nodes because of the restart caused the Datanode process hangs. While simply using the command HDFs dfsadmin-report to view and inconvenient, too much information, here is a simple script implemented with SED and awk:

Cat lastdeadnodes.sh

HDFs dfsadmin-report > All.log
#sed-n '/dead/, $p ' All.log > Deadnodes.log
Sed ' 1,/dead/d ' all.log > Deadnodes.log
Sed-i/rack/d Deadnodes.log
awk ' begin{rs= ' \n\n\n, ors= "\ n"; Fs= "\ n"; ofs= "\ t"}{print $2,$15} ' Deadnodes.log > Last.log
dt= ' Date '
Dt= ' echo $dt | Awk-f "" ' {print $ "" $ "
grep "$dt" Last.log

The script description describes the format of the data obtained by the following HDFs Dfsadmin-report:

.......

name:10.39.0.185:50010 (10.39.0.185)
hostname:10.39.0.185
Rack:/yh11070028
Decommission Status:normal
Configured capacity:46607198593024 (42.39 TB)
DFS used:22027374755910 (20.03 TB)
Non DFS used:0 (0 B)
DFS remaining:24579823837114 (22.36 TB)
DFS used%: 47.26%
DFS remaining%: 52.74%
Configured Cache capacity:0 (0 B)
Cache used:0 (0 B)
Cache remaining:0 (0 B)
Cache used%: 100%
Cache remaining%: 0%
Last contact:wed Feb 23:14:43 CST 2015

Dead Datanodes:
name:10.39.1.35:50010 (10.39.1.35)
hostname:10.39.1.35
Rack:/yh11070032
Decommission Status:normal
Configured capacity:0 (0 B)
DFS used:0 (0 B)
Non DFS used:0 (0 B)
DFS remaining:0 (0 B)
DFS used%: 100%
DFS remaining%: 0%
Configured Cache capacity:0 (0 B)
Cache used:0 (0 B)
Cache remaining:0 (0 B)
Cache used%: 100%
Cache remaining%: 0%
Last Contact:mon Jan 10:08:36 CST 2015

.......

From the above information to see in hadoop2.0 the DN node information in the "Dead Datanodes:" line, so the key to write the script is to start from this line to get all the information behind all the rows.

The following is a description of the script:

1, through the HDFs dfsadmin-report command to get the cluster so the DN information output to the All.log file, including the surviving and hanging out DN information.

2. Get the information of the DN node that is hung out, and realize it through SED

#sed-n '/dead/, $p ' all.log > Deadnodes.log #这种方式拿到的信息会包含匹配行 "Dead datanodes:"
Sed ' 1,/dead/d ' all.log > Deadnodes.log #这种方式拿到的信息是从匹配行 "Dead datanodes:" The next line to the end of the message, excluding the matching line

3, the completion of the second step after we got all the dead DN information, but after testing found that there is no DN information "Rack:" This line of information, this line is not our last need, simply delete:

Sed-i/rack/d Deadnodes.log

4, complete the third step after the information format as follows:

name:10.39.1.35:50010 (10.39.1.35)
hostname:10.39.1.35
Decommission Status : Normal
Configured capacity:0 (0 B)
Dfs used:0 (0 b)
Non Dfs used:0 (0 b)
Dfs remaining:0 (0 b)
Dfs Us ed%: 100.00%
DFS remaining%: 0.00%
Configured Cache capacity:0 (0 B)
Cache used:0 (0 B)
Cache remaining:0 ( 0 B)
Cache used%: 100.00%
Cache remaining%: 0.00%
Last Contact:mon Jan 10:08:36 CST

name:10.39.6 .197:50010 (10.39.6.197)
hostname:10.39.6.197
Decommission status:normal
configured capacity:0 (0 B)
DFS used:0 (0 b)
Non Dfs used:0 (0 b)
Dfs remaining:0 (0 b)
Dfs used%: 100.00%
Dfs remaining%: 0.00%
Con Figured cache capacity:0 (0 B)
Cache used:0 (0 B)
Cache remaining:0 (0 B)
Cache used%: 100.00%
Cache remain ing%: 0.00%
Last Contact:mon Jan 18:23:56 CST

In fact, the information we need is hostname and last contact, this step is implemented with awk, with multiple lines as a record:

awk ' begin{rs= ' \n\n\n, ors= "\ n"; Fs= "\ n"; ofs= "\ t"}{print $2,$15} ' Deadnodes.log > Last.log

Here are some of awk's key words: Rs/ors/fs/ofs

Rs:record Separator, Record delimiter
Ors:output record separate, output current record delimiter
Fs:field Separator, Field delimiter
Ofs:out of field Separator, output fields delimiter

The awk script means that the three newline character "\n\n\n" is used to process the record delimiter (rs= "\n\n\n"), from "Name:" to "last Contact:", awk is treated as a record, and the processed records are separated by a newline character (ors= "\ n") , the delimiter between the fields of one record to be processed is a newline character (fs= "\ n"), but the delimiter between the fields of one record is processed, we use \ t (ofs= "\ T"), each of which we only focus on hostname and last contact, So you only need to print the $ and $ ({print $2,$15}).

You can see the following data formats after this step:

hostname:10.39.1.35Last Contact:mon Jan 10:08:36 CST 2015
hostname:10.39.6.197 Last Contact:mon Jan 18:23:56 CST 2015
hostname:10.39.5.80 Last Contact:sat Feb 03:59:20 CST 2015
hostname:10.39.4.247 Last contact:wed Feb 17:27:51 CST 2015
hostname:10.39.6.199 Last Contact:mon Feb 10:42:21 CST 2015
hostname:10.39.7.55 Last Contact:thu Feb 00:26:17 CST 2015
hostname:10.39.0.218 Last Contact:thu Feb 07:18:54 CST 2015
hostname:10.39.0.208 Last Contact:mon Feb 12:22:13 CST 2015
hostname:10.39.4.235 Last Contact:thu Jan 08:00:00 CST 1970
hostname:10.39.4.243 Last Contact:thu Jan 08:00:00 CST 1970

5. Finally, we focus only on the DN nodes that are hung today.

dt= ' Date '
Dt= ' echo $dt | Awk-f "" ' {print $ "" $ "
grep "$dt" Last.log

Results:

hostname:10.39.7.55last Contact:thu Feb 00:26:17 CST 2015

Summary:

This script uses some special uses of awk and SED, which are documented for easy querying.

Get the latest Datanode information from the cluster with SED and awk