Linux index node (inode:no space for device) with a single failure caused by a full

Problem description

Nimbus and supervisor automatically hang on the storm test environment cluster, display no space for device when restarting, and cannot create, add files and directories, df-h view

Ilesystem            Size  used Avail  Use% mounted on/Dev/vda1               +G  2.9G    *G   8% /Tmpfs                 3.9G     0  3.9G   0%/dev/shm/Dev/VDC1              -G  3.1G    theG   4% /home/Dev/VDD1               -G   theM    -G   1%/home/xxx/hard_disk/0/Dev/Vde1               -G   theM    -G   1%/HOME/XXX/HARD_DISK/1/Dev/VDF1               -G   theM    -G   1%/HOME/XXX/HARD_DISK/2/Dev/VDG1               -G   theM    -G   1%/HOME/XXX/HARD_DISK/3/Dev/VDH1               -G   theM    -G   1%/HOME/XXX/HARD_DISK/4

Display space is used very little, enough space, df-i display

Filesystem            inodes   iused   IFree Iuse% mounted on/Dev/vda1            2621440   75783 2545657    3% /Tmpfs                1007672       1 1007671    1%/dev/shm/Dev/VDC1             102400   85159   85159   -% /home/Dev/VDD1              51200       -   51187    1%/home/xxx/hard_disk/0/Dev/Vde1              51200       -   51182    1%/HOME/XXX/HARD_DISK/1/Dev/VDF1              51200       -   51182    1%/HOME/XXX/HARD_DISK/2/Dev/VDG1              51200       -   51182    1%/HOME/XXX/HARD_DISK/3/Dev/VDH1              51200       -   51182    1%/HOME/XXX/HARD_DISK/4

found that the Inode usage rate reached 100%, which found the problem

What is Inode one, inode

The minimum storage unit for a hard disk is called "Sector" (Sector). Each sector is stored 512 bytes (equivalent to 0.5KB). When the operating system reads the hard disk, it does not read from one sector to the next, so it is too inefficient to read a "block" at a time, and each "block" consists of eight contiguous sector. This "block", composed of multiple sectors, is the smallest unit of file access. "Block" size, the most common is 4KB, file data are stored in the "block", you must also find a place to store the meta-information of the file, such as the creator of the file, the date of creation of the file, the size of the file, and so on. This area of stored file meta information is called Inode, and the Chinese name is "Index node". Each file has a corresponding inode, which contains some information about the file.

Ii. contents of the Inode

The inode contains the meta-information for the file, with the following content:
* Number of bytes in the file
* User ID of the owner of the file
* The group ID of the file
* file read, write, execute permissions
* File timestamp, total three: CTime refers to the time when the inode was last changed, mtime refers to the time when the file content was last changed, atime refers to the time when the file was last opened.
* Number of links, that is, how many filenames point to this inode
* Location of File data block
You can use the Stat command to view inode information for a file:

[[Email protected] home]# stat AppsFile: ' Apps ' Size:4096Blocks:8IoBlock:4096directorydevice:fd02h/64770D Inode:4123Links:3Access: (0700/drwx------) Uid: (456/apps) Gid: (456/apps)Access: --Geneva- - Geneva: Geneva:07.069500045+0800Modify: --Geneva- -  the: *:57.886985112+0800Change: --Geneva- -  the: $:28.209482045+0800

Third, the size of the Inode

The inode also consumes hard disk space, so when the hard disk is formatted, the operating system automatically divides the hard disk into two zones. One is the data area, the file data is stored, and the other is the Inode area (inode table), which holds the information contained in the Inode.
The size of each inode node is typically 128 bytes or 256 bytes. The total number of inode nodes, given at the time of formatting, is usually set to one inode per 1KB or 2KB each. Assuming that the size of each inode node in a 1GB hard disk is 128 bytes, and one inode is set per 1KB, the Inode table will be about 128MB in size, accounting for 12.8% of the entire drive.
Use Df-i to see the total number of inode and the number of uses for each hard disk partition

Solutions

Go to the home path and use Ls-i to see what information is used for each path:

    inxxxxxxxx83969xxxxxxxx79877xxxxxxxx -curator-recipes-2.5. 0. Jar        OneLost+found86017xxxxxxxx theCurator. Tar. GZ               90113Mapred88065xxxxxxxx94209xxxxxxxx79873Mobilereco98305xxxxxxxx theFds. Jar                          -Ro_test81921Vpp69633Hdfs73729Spark20481Yarn92161xxxxxxxx96257xxxxxxxx129xxxxxxxx8193Storm

The number of inode used for the pavement will be displayed in front of each directory

How to release inode information, view the following information

It's quite easy for a disk to has a large number of inodes used even if the disk is not very full.
An inode was allocated to a file so, if you have gazillions of files, all 1 bytes each, and you'll run out of Inodes long befor E You run out of disk.
It's also possible that deleting files would not reduce the inode count if the files has multiple hard links. As I said, inodes belong to the file, not the directory entry. If a file has both directory entries linked to it, deleting one won't free the inode.
Additionally, you can delete a directory entry but, if a running process still have the file open, the Inode won ' t be Freed.
My Initial advice would is to delete all the files can, then reboot the box to ensure no processes is left Holdi ng the files open. The
If you do and still has a problem, let us know.
by the the-the-looking for the directories-contain lots of files, this script could help:

#!/bin/bash# count_em - count files in all subdirectories under current directory.echo‘echo $(ls -a "$1" | wc -l) $1‘700 /tmp/count_em_$$find . -mount -type d -print0 | xargs -0-f /tmp/count_em_$$

Or

.-xdev-type|-d"/"-f2||-c|-n

The list the files and remove these files to free inode.

Reference documents
1. Howto Free Inode Usage
2. A single failure caused by a full Linux index node (Inode)
3. Understanding of the Inode of Linux

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Linux index node (inode:no space for device) with a single failure caused by a full