Business process restart failed due to inode exhaustion under UFs file system

Once the business upgrade, found that the business process on the production system UPRG can not start, the log error: Can not create UPRG.log, but the observation/logs directory remaining space there are many. Try to directly under the/logs touch file also failed, also reported can not create file.

The first response is whether the permissions of the/logs directory have been changed by mistake? But soon the directory permissions were found to be fine.

The second reaction is that the disk is broken, but think of the disk is RAID1, it is not possible to two disks are bad, and the system log does not have any disk error.

The third reaction is that the partition table is broken, but if the partition table is broken, the CD should not go in.

。。。

Finally found that the inode was exhausted, with Find+xargs crazy Delete a heap of log files, the business process started successfully.

(The following reference http://blog.chinaunix.net/uid-119039-id-2802580.html)

Inode Number Calculation formula:
inode_number= File System Size/NBPI

File system size (GB) default NBPI size (byte)
≤1 2048
1<fs≤2 4096
2<fs≤3 6144
3<fs≤1023 8192 8K
≥1024 (i.e. 1T) 1M

Our/logs is 64G, so the total inode count is 64*1024*1024/8=800 million.

Which means /logs the directory can only be created more than one file, and this production machine on a new business, each user's log is generated by a separate log file, This results in a spike in the number of small files, which drains the inode during the log cleanup cycle, causing problems.

In addition, the feeling of Solaris is not human-friendly, in this case, the report: Can not create file, it can be reported:

Can not create the file due to exhaustion of inode. Even for many years of operation, it is difficult to run into a production system where the inode is exhausted, and in an emergency of business recovery, it is still very crazy.

This article is from the "Memory Fragments" blog, so be sure to keep this source http://weikle.blog.51cto.com/3324327/1617091

Business process restart failed due to inode exhaustion under UFs file system