Linux server disk space is full.

Linux server disk space is full.
Next, let's take a look at the article about how to solve the problem of disk occupation of Linux servers (/dev/sda3 is full). We hope that people who encounter such problems can help us.

Today, leader of an e-commerce technology department found a problem, saying that their server hard disk is full. Delete all log files, but the hard disk space is still full. Therefore, df-h checks the status of each mount point (for example ).

/Dev/sda3 occupies 100%, so we can view the directory usage (for example) in du-s-h ).

 

 

During work, we may encounter this problem and find that a disk space is almost full. Therefore, after we find some useless large files and delete them, we find that the disk space has not been released, why? How can this problem be solved? Next we will reproduce the entire process:

View disk space

[@ 74.114 var] # df-h
Filesystem Size Used Avail Use % Mounted on
/Dev/xvda2 9.7 GB 284 M 8.9G 4%/
/Dev/xvda1 251 M 13 M 226 M 6%/boot
None 1.1G 0 1.1G 0%/dev/shm
/Dev/xvda10 97G 60G 33G 65%/home
/Dev/xvda3 3.9G 2.7G 1.1G 72%/usr
/Dev/xvda5 3.9G 3.6G 77 M 98%/var

The var partition is almost full. Find the large file and delete it.

[@ 74.114 var] # cd/var
[@ 74.114 var] # du-sh *
3.3G account
111 M cache
53 M log
0 mail
156 K run
344 K spool
...
[@ 74.114 var] # rm-rf account /*

Df-h: Check whether/var is 98% or not.

[@ 74.114 var] # df-h
Filesystem Size Used Avail Use % Mounted on
/Dev/xvda2 9.7 GB 284 M 8.9G 4%/
/Dev/xvda1 251 M 13 M 226 M 6%/boot
None 1.1G 0 1.1G 0%/dev/shm
/Dev/xvda10 97G 60G 33G 65%/home
/Dev/xvda3 3.9G 2.7G 1.1G 72%/usr
/Dev/xvda5 3.9G 3.6G 77 M 98%/var

However, du-sh * shows that there are no large files.

[@ 74.114 ~] # Cd/var & du-sh *
120 K account
111 M cache
53 M log
0 mail
156 K run
344 K spool
...
Conjecture 1: The deleted content is still occupied by the process and the memory is not released. Therefore, use lsof | grep-I deleted to check it and find the following:

[@ 74.114 account] # lsof | grep-I deleted
Listserve 4833 blty 0u CHR 136,2 4/dev/pts/2 (deleted)
Listserve 4833 blty 1u CHR 136,2 4/dev/pts/2 (deleted)
Listserve 4833 blty 2u CHR 136,2 4/dev/pts/2 (deleted)
Billing_P 16989 blty 0u CHR 136,0 2/dev/pts/0 (deleted)
Billing_P 16989 blty 1u CHR 136,0 2/dev/pts/0 (deleted)
Billing_P 16989 blty 2u CHR 136,0 2/dev/pts/0 (deleted)
Billing_P 16990 blty 0u CHR 136,0 2/dev/pts/0 (deleted)
Billing_P 16990 blty 1u CHR 136,0 2/dev/pts/0 (deleted)
Billing_P 16990 blty 2u CHR 136,0 2/dev/pts/0 (deleted)

The billing program was not released at the time of discovery. Therefore, contact the project manager to stop Billing_P and restart the program. Fruitless
Conjecture 2: The deleted file is the pacct file under the accout directory, which should be generated and managed by psacct. Will it be OK after the program is restarted? Solution

[@ 74.114 account] #/etc/init. d/psacct restart

Finally, let's take a look at the disk space. The disk space is released ~~

[@ 74.114 var] # df-h
Filesystem Size Used Avail Use % Mounted on
/Dev/xvda2 9.7 GB 284 M 8.9G 4%/
/Dev/xvda1 251 M 13 M 226 M 6%/boot
None 1.1G 0 1.1G 0%/dev/shm
/Dev/xvda10 97G 60G 33G 65%/home
/Dev/xvda3 3.9G 2.7G 1.1G 72%/usr
/Dev/xvda5 3.9G 100 M 3.6 M 4%/var

TIPS:

If you find that the size of the du and df is inconsistent in the future, you can use lsof to check the cause of the problem. If you still cannot find the problem, if allowed, try restarting the service. The problem may be solved.
If the service cannot be restarted, You Can unmount the disk partition to solve the problem.
Before deciding to delete certain files, it is best to determine which services are used for this file, stop these services and then delete them, so that the space will rarely be released.

 

Linux disk mount point directory usage (figure)

The directory and far from the mount point are smaller than 4.5 GB. What occupies the hard disk?

The following are the answers to this question:

When the apache/tomcat service is running, logs of the running service are cleared, resulting in/dev/sda3 being full. Generally, most services (including scripts) cannot delete the log files currently being written at runtime.

Principle Analysis:

1. The current access. log is being occupied by the apache process. 2. Use the rm command to delete access. log. Actually, only the file name is deleted (the log file's application count is not 0, so the space will not be released ). 3. the access. after logging, apache still writes logs to access. log, when the apache process is enabled, the access. log located the inode of the file, that is, the log is not written through access. log, even if the access. log, apache still writes logs to access. the inode node where the log is located, resulting in increased hard disk space. 4. Because access. log is deleted, we cannot find the file, and du cannot find it. The hard disk is full, but we cannot see which files are occupied.

(If you do not know the details, you can google them again .)

Solution:

Restart the related services or programs of the log file. For example, restart apache if the log file is an apache Log File (for example, hard disk space usage is normal after restart ).