Delete files in Linux, disk space not released issue tracking

After customers use our products, they find a problem: Disk space is not released after the file has been deleted. Is there a process in opening this file, or is there something else? Let's take a look at two scenes

I. Scenario ONE: The process opens this file

When a file is being used by a process, the user deletes the file, and the file is only removed from the directory structure, but is not deleted from the disk. When the process of using this file ends, the file is actually deleted from the disk, freeing up the occupied space.

We found that the remaining disk space is less, go back to delete some large temporary files or log files, if the deletion will find that the disk space is not reduced, you can use the "lsof" command to see the process using the file, and then restart the process or service.

"Example"

The disk space is now found to occupy 99%, leaving only 522M of space left.

Suse11x64-001:/test # Df-hfilesystem                       Size  used Avail use% mounted on/dev/sda2                         29G   27G  522M  99%/devtmpfs                         972M  116K  972M   1%/devtmpfs                            972M     0  972M   0%/DEV/SHM

Found a file "Vmcore" occupies close to 900M space, but this file does not need to use, so the "rm" command to delete this file, but after deletion, found that disk space is not really reduced.

Suse11x64-001:/test # rm Vmcore suse11x64-001:/test # df-hfilesystem                       Size  used Avail use% mounted on/dev/sda2< c18/>29g   27G  522M  99%/devtmpfs                         972M  116K  972M   1%/devtmpfs                            972M     0  972M   0%/dev/shm//10.204.16.2/home/splx/iceking  6.3T  1.6T  4.7T  25%/mnt/iceking

This means that there is a good chance that other processes are using this file, using the " lsofcommand to see the process that is using the file.

Suse11x64-001:/test # Lsof | grep vmcorea.out      2610       root    3r      REG                8,2 941331144    1643779/test/vmcore (Deleted)

Process number is 2610(The process is named " a.out") process, you are using the Vmcore file, and you can see that there are" deleted": It indicates that the file being used was deleted, but was not actually removed from the disk.

Now we remove this process and view disk space at this time the occupancy rate is reduced to 95% and the remaining space is increased to 1.4G.

Suse11x64-001:/test # Df-hfilesystem                       Size  used Avail use% mounted on/dev/sda2                         29G   26G  1.4g< c16/>95%/devtmpfs                         972M  116K  972M   1%/devtmpfs                            972M     0  972M   0%/DEV/SHM

Two. Scenario two: Kernel module bug

The information required to process files in the file system is stored in the index node (inode), and if the reference count of the index node is not 0 (indicating that the file is being used) when the file is deleted, the file is not actually deleted on disk, ensuring that the process that is using the file is working properly with the file.

First, let's take a look at some of the key data structures associated with the file system in the kernel, and when a process opens a file, it creates an object in the kernel that describes how the process interacts with the file. The file object will point to a dentry structure (catalog Item) that describes the catalog item name, parent directory entry information, subdirectory item information, and so on. The inode node that d_inode in dentry points to contains the information that the actual file is stored on disk.

    When multiple processes open the same file, the kernel changes to create the corresponding file object, but they all share the same dentry , except every time you open the file dentry reference Count d_count Plus 1. And for the same file that is opened, inode is also unique, inode The reference count of i_count is typically the number of hard links to a file. read some Chinese blogs and say "the same file, every time you open it, refers to the count i_count plus 1 ", which is wrong by my validation results. The experimental result is that the reference count in inode does not change for the same file every time it is opened, but the corresponding dentry reference count plus 1.

After the client deleted the file, the disk space was not released and the process that was occupying the file was not found through the "lsof" command. Again, this is due to the early formation of the kernel module of the product. After analysis: In the previous blog post "Linux kernel module memory leak lookup (2)" explained that because in the product kernel module, the dentry reference, and after the use of its reference count minus 1, resulting in a memory leak. In this case, thedentry is not released, and the inode is always referenced, causing the deletion of the file even if it is deleted.

And for the above problems and analysis, if not timely to the customer fix this problem, it can only let it restart the OS, free disk space will be released.