Linux disk space Surge all records of sudden troubleshooting Process

The entire process of resolving sudden Linux disk space spikes-Linux Enterprise Application-Linux server application information. The following is a detailed description. 1. In the afternoon, the user reported that files could not be printed to the network printer through a server. other machines are normal.
Error:
Lpr: error-unable to print file: client-error-request-value-too-long

2. Cause of error:
1. Are you trying to print a file> 2 GB? If so, that doesn' t
Work in CUPS 1.1.x and earlier.

2. Does the RequestRoot directory (/var/spool/cups by default)
Exist? If not, "mkdir/var/spool/cups"

3. Does the TempDir directory (/var/spool/cups/tmp by default)
Exist? If not, "mkdir/var/spool/cups/tmp"

4. Is the disk full? "Df-k/var/spool/cups" will show if
This is the case. If the disk is full, delete files
Free up disk space.

3. identify the cause of the fourth point
Df-h,
/Var 100% used.

4. Positioning
Cd/var
Find.-maxdepth 1-type d-print | xargs du-sk | sort-rn
Locate the file
/Var/log/bandwidth occupies 19 GB Space

5. Delete
Rm-rf bandwidth

6. the space is still not cast

7. Check the cpu. The rotate process occupies a large amount of cpu,
Kill-9

8. It is restored to normal and can be printed.

9. Continue to find the reason
Find/home-type f-ls | perl-e 'while (<>;) {$ s + = (split) [6] ;}; print "$ s \ n ";'

This will count all hidden files, including the ones from lost + found.

The number shocould be close to what how much space is used on disk.

If df shows a something way different you may be want to run something like:
Lsof | grep home

Look for some suspicious applications.

The idea is that if an application opens a file and the file is removed
While the application keeps it open, the actual data is not removed from
Disk until the program exits or close the file.

My advice:
1. boot in single mode, recommended from a rescue disk so you have
'Clean' kernel.
2. test du/find versus df output.
3. if different run fsck with-f and, if you can afford to wait,-c flags.
4. I think du/find shocould show similar numbers now (if it doesn' t and
You booted with a 'clean' kernel then I'll be interested for details)
5. reboot with the normal kernel
6. if df shows different than du then most likely you 've been hacked-
Replace OS with a clean copy-try to find what programs are different
Than the original, etc.

Delete the file. If the process is running, the space will not be released.

10. Check for/etc/syslog. conf
In
Kern. = debug-/var/log/bandwidth
Automatically configured when webmin is installed.

Comment out. Everything is fine.