Thoughts and implementation of disk scripts in the big data environment

Requirements:

Consider a scenario, such as the disk required by the storage node (datanode) in hdfs. When the cluster size increases, assume that each node has 20 disks and the cluster has 200 nodes, so the actual physical disk is 4000 disks. Assuming that the annual failure rate of the disk is 1%, can we consider that one disk needs to be replaced or repaired every day when it fails?

Now, in order to free up productivity, the specific requirement is: after a disk failure is discovered, you only need to replace the physical disk, and all system-level operations are automated.

Some questions:

1): Delete the faulty disk information written to fstab.

2): The new disk does not need to be partitioned and is formatted directly.

3): If the faulty disk is restored, it is retained. If the disk data is retained, it cannot be formatted.

4): You cannot operate disks with partitions.

5): cannot operate on mounted disks (considering hot swapping of faulty disks)

6) assume that the total number of disks is uncertain. Therefore, the number of mount points should correspond to the number of disks.

7): You cannot format available data.

8): the premise for disk operations must be that the disk has no partitions and no data.

9): mount the formatted but not mounted disk.

10): for data security, please add...

System Environment: CentOS 6.3 x86_64

The shell code is as follows:

#! /Bin/bash # Script name: check_disk.sh # Date & Time: 2013-07-08/21: 06: 08 # Description: # Author: MOSMlog =$ {Mlog: -/var/log/check_disk.log} Date = "/bin/date + % k: % M: % S/% Y-% m-% d "NUL =/dev/null # The parent directory of the mount point. The SDIR =/mpoint # Umt function is automatically created: unmount the invalid disk Umt () {Ut = ('grep "$ SDIR"/etc/fstab | awk '{print $2 }'') ud = ('grep "$ SDIR"/etc/fstab | awk-F \ "'{print $2}') for d in $ {Ut [@]}; do if [-d $ d]; then # Dk = 'df $ d | awk '{print $1}' |/u Sr/bin/tail-1 'dk = 'mount | grep "$ d \>" | awk '{print $1}' Ud = '/sbin/blkid $ Dk | awk-F \ "'{print $2}'' Td = 'grep $ d/etc/fstab | awk-F \ "' {print $2}'' [-B "$ Dk"] & Uft = '/sbin/parted $ Dk print |/usr/bin/tail-2 |/usr/bin/head-1 | awk' {print $5} ''tft = 'grep $ d/etc/fstab | awk '{print $3} ''if [[$ Ud! = $ Td | $ Uft! = $ Tft]; then/bin/umount $ d &> $ NUL/bin/sed-I "/$ d/d"/etc/fstab &> $ NUL fi done for d in $ {ud [@]}; do Unull = '/sbin/blkid | grep $ d' if [-z $ Unull]; then Mpit = 'grep $ d/etc/fstab | awk '{print $2} ''/bin/umount $ Mpit &> $ NUL/bin/sed-I"/$ d/d "/etc/fstab &> $ NUL fi done} # Mmu function: mount valid disk Mmu () {declare-I P = 1 Tb =/etc/fstab Ftb () {Sft = '/sbin/blkid $ Fph | awk-F \ "' {print $2} 'Ft = 'grep" \ <disk $ P \>" $ Tb 'gud = 'grep "\ <$ Sft \>" $ tb' [[-n $ Ft] &/bin/sed-I "/\ <disk $ P \>/"d $ Tb [[-n $ Gud] &/bin/sed-I"/\ <$ Sft \>/"d $ Tb} Add () {Ef = '/sbin/blkid $ fph' [[-z $ Ef] &/bin/echo y |/sbin/mkfs. ext4 $ Fph &> $ NUL Uuid = '/sbin/blkid-s UUID $ Fph | awk' {print $2} ''Fs ='/sbin/blkid-s TYPE $ fph | awk-F \ "'{print $2}'' Fr = "defaults, noatime "echo-en" $ Uuid \ t "$ SDIR"/disk $ P \ t $ Fs \ t $ Fr \ t0 0 \ n ">>$ Tb/bin/m Ount-a &> $ NUL break} Fe () {Mtp = 'mount | grep "" $ SDIR "/disk $ P \>" 'if [[-n "$ Mtp"]; then echo "'$ date' $ Mp is used. ">>$ Mlog else Ftb Add fi} while:; do if [[-d" $ SDIR "/disk $ P]; then Fe else/bin/mkdir-p "$ SDIR"/disk $ P & Ftb Add fi P = P + 1 done} # Chech_disk function: differentiate disk validity and call the corresponding function to process Check_disk () {[-x/sbin/partprobe] & amp;/sbin/partprobe & gt; $ NUL Umt/bin/mount-a & gt; $ NUL for B in $ {Exist [@]}; do Fp H = "$ Dph $ B "[[! -B "$ Fph"] & echo "'$ date' $ Fph is not block device !!! ">>$ Mlog & continue CTE = 'ls" $ Fph "* |/usr/bin/wc-L' [[[" $ CTE "-ne 1] & echo "'$ date' $ Fph mismatches. ">>$ Mlog & continue DFM = 'mount | awk' {print $1} '| grep" ^ $ Fph $ "2> $ NUL' [[-n" $ DFM "] & echo" '$ date' $ Fph is used. ">>$ Mlog & continue Ugd = '/sbin/parted $ Fph print |/usr/bin/wc-L' [[" $ Ugd "-ge 8 &&- n "$ DFM"] & echo "'$ date' parted $ Fph> = 8. ">>$ Mlog & continue Lvm = '/sbin/blkid $ Fph | grep-E" LVM2_member | swap "' [[-n" $ Lvm "] & echo "'$ date' $ Fph is LVM or swap. ">>$ Mlog & continue Fgd = '/sbin/fdisk-l $ Fph |/usr/bin/wc-L' [[" $ Fgd "-ge 10] & echo "'$ date' fdisk-l $ Fph> = 10. ">>$ Mlog & continue Mmu done} if [[" $1 "="-s "&-z" $2 "]; then if [[-B/dev/cciss/c0d0]; then Ah = hp Dph =/dev/cciss/Exist = ('ls/dev/cciss/| grep "^ \ <c0d [0-9] [0-9] \ {0, 1 \} \> "') check_disk else Exist = ('ls/dev/| grep-E "^ \ <sd [a-z] {0, 1} \> | ^ \ 




Remember to add-s for use in the format of./check_disk.sh-s. The script can be downloaded directly as an attachment.
It's not easy to write things. I think it's just a compliment. Thank you.


This article from "self-improvement" blog, please be sure to keep this source http://mos1989.blog.51cto.com/4226977/1243650