Solution: No spaces left on device

Find out why

In a literal sense, the problem is that there is no extra space on the disk .

So exactly where is the space. Check the current calculator disk idle with the DF command first

Df-a

When I'm done here, I can see that/dev/vda1 is completely occupied. Start with the du command to find the most space-consuming files from the root directory

#View the space occupied by each folder in the current directory
Du-sh *

Through a layer of comparison files occupy space, found that the Jenkins run log files occupy the largest, originally my side of the server disk only 100G, the result of this log file has 86G, I understand that this file will be filled with the disk solution

My first thought was to delete the file directly, so I went into the Jenkins directory and deleted all the logs directly.

RM-RF *.log

After the command is executed, other programs on the server are temporarily functional. But when I ran the DF command, I found that/dev/vda1 was still fully occupied, but I thought that the application was running normally, and that maybe the system needed to reboot or take some time to update the state, and I didn't bother with it.

However, this issue has not ended. The next day my colleague discovered that the application was inaccessible and I realized that the problem was not over. I performed the DF and Du commands again, and found that the results of DF indicated that/DEV/VDA1 was still fully occupied, while the results of Du did not find any larger files. I tried to solve the problem of inconsistency between the two results, perhaps resolved to solve the "No spaces left O device" problem.

Based on two command results, I guess the file space that was deleted yesterday directly from the RM command was not freed, so I looked at all the files that were open in the system and found the log file "Jenkins.log".

lsof | grep "Jenkins.log"

many processes in the list open the file, although the file was deleted, but the process that opened the file did not close, meaning that the file actually existed, and RM simply deleted the file's markup .

So I decisively terminated the process that opened the file.

Kill-9 22731

After the execution, again running DF and Du, found that the results of the difference is not big. To this system again can run normally.

Although this problem is a temporary solution, but the Jenkins log why can reach 86G. This problem has not been resolved, because this time the deletion of the full, did not take a closer look at the contents of the log, the cause of the problem is not too able to detect. But the root of the problem is certainly not resolved, can only wait until the next time the problem appears, a good study of the log, fundamentally solve this problem.

Related information 

CentOS How to clean up the inconsistency between/DEV/VDA1 system disk DF and du LOSF command description