Hadoop cluster space usage report script

The cluster space has been a little tight recently and is always worried about space shortage and crashes. The recent resizing is not realistic. After communicating with cluster users, we found that the cluster stores a lot of useless historical data and can be deleted, in this way, you can use a crontab script to generate a cluster space usage report every day. When the usage exceeds 70% and 80%, alarms are reported separately, and users are notified that the occupied space is large and the redundant space is reserved, in this way, you do not need to worry that the cluster space is full.

[Hdfs @ hanagios48 root] $ more/home/hdfs/dfsadmin_report.sh
#! /Bin/bash
Source ~ /. Bash_profile

Today = $ (date + % Y % m % d)
# Report = 'hadoop dfsadmin-report | head-n 11'

Report =/tmp/report
Echo "Hi, cluster users
When the Hadoop cluster space usage reaches 70%, you will receive a Warning email prompting you to clear the space;
When the usage of the Hadoop cluster space reaches 80%, you will receive a Critical email prompting you to clear the space;
To ensure cluster data security and reserve down-machine redundancy space, please be sure to clean up. If the cluster cannot accommodate a large amount of data, please inform the O & M personnel for resizing in time. Thank you! "> $ Report

Echo> $ report
Hadoop dfsadmin-report | head-n 11 >>$ report

Echo ----------------------> $ report
Echo dfs used details: >>$ report
Hadoop fs-du/> $ report
Echo> $ report
Hadoop fs-du/user> $ report


Dfs_used_percent = 'cat $ report | grep "DFS Used %" | awk-F: '{print $2 }''
Dfs_used = 'echo $ {dfs_used_percent} | awk-F % '{print $1 }''


# Decimal comparison
# Expr $ {dfs_used }\>\= 80

User = laijingli2006@126.com
Title = 'echo "$ {today} [$ {dfs_used_percent}] WBY Hadoop Cluster Hdfs Useage Report: dfs_used $ {dfs_used_percent }"'
Echo $ title


If [$ (expr $ {dfs_used }\>\= 80) = 1]; then
Echo dfs_used 80
# Cat $ report | mail-s "Critical: $ title" $ user
Cat $ report |/usr/bin/mutt-s "Critical: $ title" $ user

Elif [$ (expr $ {dfs_used} \> \ = 70) = 1]; then
Echo 70
# Cat $ report | mail-s "Warning: $ title" $ user
Cat $ report |/usr/bin/mutt-s "Warning: $ title" $ user

Else
Echo 60
# Cat $ report | mail-s "Normal: $ title" $ user
# Cat $ report |/usr/bin/mutt-s "Normal: $ title" $ user
Fi

# Cat $ report | mail-s $ today DfsReport: dfs_used $ {dfs_used_percent} 362560701@qq.com

Crontab execution is good:

[Hdfs @ hanagios48 root] $ crontab-l
05 8 ***/home/hdfs/dfsadmin_report.sh

[Hdfs @ hanagios48 root] $ more/tmp/report
Hi, cluster users
When the Hadoop cluster space usage reaches 70%, you will receive a Warning email prompting you to clear the space;
When the usage of the Hadoop cluster space reaches 80%, you will receive a Critical email prompting you to clear the space;
To ensure cluster data security and reserve down-machine redundancy space, please be sure to clean up. If the cluster cannot accommodate a large amount of data, please inform the O & M personnel for resizing in time. Thank you!


Configured Capacity: 124854950621184 (113.55 TB)
Present Capacity: 118317151626783 (107.61 TB)
DFS Remaining: 38704545865728 (35.2 TB)
DFS Used: 79612605761055 (72.41 TB)
DFS Used %: 67.29%
Under replicated blocks: 0
Blocks with primary upt replicas: 0
Missing blocks: 0


-------------------------------------------------
Datanodes available: 15 (15 total, 0 dead)
----------------------
Dfs used details:
Found 6 items
0 hdfs: // hamaster140: 9000/benchmarks
125752 hdfs: // hamaster140: 9000/data0
0 hdfs: // hamaster140: 9000/system
13721821810608 hdfs: // hamaster140: 9000/tech
1803375805154 hdfs: // hamaster140: 9000/tmp
6411197575455 hdfs: // hamaster140: 9000/user

Found 14 items
33222938 hdfs: // hamaster140: 9000/user/azk
4072247213805 hdfs: // hamaster140: 9000/user/CIA
40705761240 hdfs: // hamaster140: 9000/user/din
0 hdfs: // hamaster140: 9000/user/fea
0 hdfs: // hamaster140: 9000/user/gao
36454169547 hdfs: // hamaster140: 9000/user/gmz
1877816487439 hdfs: // hamaster140: 9000/user/hdf
148965233376 hdfs: // hamaster140: 9000/user/imp
2416017438 hdfs: // hamaster140: 9000/user/in
0 hdfs: // hamaster140: 9000/user/lin
0 hdfs: // hamaster140: 9000/user/luo
149973222708 hdfs: // hamaster140: 9000/user/shi
82586246964 hdfs: // hamaster140: 9000/user/wuy
0 hdfs: // hamaster140: 9000/user/zho

[Hdfs @ hanagios48 root] $