Linux large files have been deleted, but df does not reduce the used space.

Linux large files have been deleted, but df does not reduce the used space.
Linux large files have been deleted, but df does not reduce the used space.

When the disk is full, you are about to delete some large files.

So I used ncdu to check the files that are currently occupying a large amount of resources in the system. If they are at the source, delete them,

Check the disk situation with joy.

The disk does not change much.

The problem is not terrible. Find baidu/google and find the cause, saying that resources have been deleted but not released.

I found some information from the Internet:

Execute lsof | grep deleted. I found that a large number of processes that have just deleted files exist, and I will kill them. But it may have some impact. For example, if you delete a file that is being written, the deletion method is incorrect.

1. Error

The O & M monitoring system sends a notification, reporting that the space of one server is full. log on to the server and check that the root partition does not have space, as shown in:

This section describes some server deletion policies. Because Linux does not have the recycle bin function, all files to be deleted on our online server will first be moved to the/tmp directory of the system, then, the data in the/tmp directory is cleared regularly. This policy is correct, but check shows that the system partition of this server is not separately divided into/tmp partitions, so the data in/tmp actually occupies the space of the root partition. Now that the problem is found, delete some big data in the/tmp directory and run the following command to check the maximum three data files in/tmp, as shown in:

1234	[Root @ localhost ~] # Du-s/tmp/* | sort-nr | head-369206016/tmp/access_log36/tmp/heat erfdata_root36/tmp/heat erfdata_mapred

Command output shows that there is a GB file access_log in the/tmp directory. This file should be an access log file generated by apache. From the log size perspective, the apache log file has not been cleared for a long time. It is basically determined that the root space is full due to this file. After confirming that this file can be deleted, perform the following delete operations:

[Root @ localhost ~] # Rm/tmp/access_log

Check whether the system root partition space is released, as shown in:

The output shows that the root partition space is still not released. Why?

2. Solution

In general, there will be no cases where the space will not be released after the file is deleted, but there are exceptions, such as the file is locked by the process, or a process has been writing data to the file, etc. To understand this problem, you need to know the file storage mechanism and Storage Structure in Linux.

A file is stored in the file system in two parts: the data part and the pointer part. The pointer is located in the meta-data file system. After the data is deleted, this pointer is removed from meta-data, and the data is stored in the disk. After the pointer corresponding to the data is cleared from meta-data, the space occupied by part of the file data can be overwritten and new content can be written. After the access_log file is deleted, the space has not been released, it is because the httpd process is still writing content to this file, so although the access_log file is deleted, the pointer of the file is not cleared from meta-data because the process is locked, since the pointer has not been deleted, the system kernel considers that the file has not been deleted. Therefore, it is not surprising that the query space through the df command has not been released.

3. troubleshooting

Now that you have the idea of solving the problem, let's see if there are any processes that are always going to acess. the lsof command in Linux is used to write data in log files. You can use this command to obtain a list of files that have been deleted but are still occupied by the application. The command is executed as follows:

The output result shows/tmp/acess. the log file is locked by the httpd process, and the httpd process has been writing log data to the file. From the seventh column, we can see that the log file size is only 70 GB, the total size of the system root partition is only 100 Gb. As a result, this file is the culprit that causes space depletion of the system root partition. In the "deleted" status in the last column, this log file has been deleted, but the space is not released because the process is still writing data to this file.

4. Solve the Problem

The problem has been fixed here. There are many ways to solve this problem. The simplest way is to shut down or restart the httpd process. Of course, you can also restart the operating system, however, this is not the best method. The best way for such processes to constantly write logs to files and release the disk space occupied by files is to clear the files online, you can run the following command:

[Root @ localhost ~] # Echo "">/tmp/acess. log

In this way, the disk space can be released immediately, and the process can continue to write logs to files. This method is often used to clean up log files generated by Web services such as Apache, Tomcat, and Nginx online.