[LINUX]DF disk 100%used alarm, du display directory status good troubleshooting

1. Review:

An online host late-night continuously received alarm system disk used overrun alarm.

Login host View But encountered confusion: in check disk usage df–h out of the disk usage is exactly the same as the alarm information, has been occupied by 100%, but to view the directory Size Du, but show the actual directory size is not the case, but there is a lot of free space.

Disk usage df–h result is 100%used,

The actual total size of the directory du–h–max-depth=1, showing the total directory size of 60k, almost negligible usage scale.

The "Knowledge preparation" [Linux] lsof command understands:

Lsof (list open files) is a tool to view the current system files. In a Linux environment, everything is in the form of files, with files that not only access regular data, but also access to network connectivity and hardware. such as Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) sockets, the system assigns a file descriptor to the application in the background, which provides a lot of information about the application itself.

Lsof open files can be:

1. Normal file

2. Directory

3. File for Network File system

4. Character or device file

5. (function) Shared library

6. Pipes, Named Pipes

7. Symbolic Links

8. Network files (for example: NFS file, network Socket,unix domain name socket)

9. There are other types of files, etc.

"Man Linux" results: Contains a large number of parameters:

[email protected] local]# man lsof
LSOF (8) LSOF (8)

NAME
Lsof-list Open files

Synopsis
lsof [-?ABCHLNNOPRTUVVX] [-A A] [C-C] [+c c] [+|-d d] [+|-d d] [+|-e s] [+|-f [CFGGN]] [-F [f]] [
-G [S]] [-I [i]] [k K] [+|-l [L]] [+|-m m] [+|-m] [-o [O]] [-P S] [+|-r [T[m<fmt>]] [-S [p:s]] [
-S [t]] [-t [t]] [u S] [+|-w] [-X [FL]] [-Z [z]] [-z] [--] [names]

Command parameters

-a lists the processes that exist for open files

-c< Process name > List files opened by the specified process

-G list GID process details

-d< File Number > list the process that occupies the file number

+d< directory > List files that are open under directory

+d< directory > recursively list files opened in directory

-n< directory > List files that use NFS

-i< conditions > lists the processes that meet the criteria. (4, 6, protocol,: Port, @ip)

-p< Process number > List files opened by the specified process number

-U list UID number process details

-H Display Help information

-V Display version information

Actual use:

$lsof | Morecommand     PID      USER   FD      TYPE             DEVICE size/off       NODE nameinit          1      root  cwd       Dir              253,0     4096          2/init          1      root  rtd       DIR              253,0     4096          2/init          1      root  txt       REG              253,0   150352    1310795/sbin/initinit          1      root  mem       reg              253,0    65928    5505054/lib64/libnss_files-2.12.soinit          1      root  mem       REG              253,0  1918016    5521405/lib64/libc-2.12.soinit          1      root  mem       REG              253,0    93224    5521440/lib64/libgcc_s-4.4.6-20120305.so.1init          1      root  mem       REG              253,0    47064    5521407/lib64/librt-2.12.soinit          1      root  mem       REG              253,0   145720    5521406/lib64/libpthread-2.12.so ...

Description

The meaning of the lsof output column information is as follows:

• COMMAND: Name of the process

• PID: Process Identifier

• PPID: Parent Process Identifier (the-R parameter needs to be specified)

• USER: Process Owner

• Pgid: The group to which the process belongs

• FD: File descriptor in which the application recognizes the file through a file descriptor. such as CWD, TXT, etc.:

(1) CWD: Indicates the current working dirctory, which is the directory in which the application starts, unless it makes changes to the directory itself (2) txt: The type of file is program code, such as the application binary file itself or a shared library, The/sbin/init program shown in the list above (3) lnn:library references (AIX), (4) ER:FD information error (see NAME column), and (5) Jld:jail directory (FreeBSD); (6) ltx:shared Library text (code and data), (7) Mxx:hex memory-mapped type number XX. (8) M86:dos Merge mapped file ; (9) mem:memory-mapped file; (ten) mmap:memory-mapped device; (one) pd:parent directory; () Rtd:root directory; Tr:kernel trace File (OpenBSD) v86 vp/ix mapped file; (15) 0: Standard input (16) 1: Standard output (17) 2: Indicates standard error generally in standard output, standard error, The standard input is followed by the file state mode: R, W, U, etc. (1) U: Indicates that the file is open and is in read/write mode (2) r: Indicates that the file is open and is in read-only mode (3) W: Indicates that the file is open and is in (4) space And no Lock (5)-: Indicates that the file's state mode is Unknow, and is locked at the same time after the file state mode, followed by the relevant lock (1) N: forA Solaris NFSLockof unknown type; (2) R: forReadLockOn part of the file; (3) R: forA readLockon the entire file; (4) W: forA writeLockOn part of the file; (partial write lock of files) (5) W: forA writeLockon the entire file; (write lock for the entire document) (6) U: forA read and writeLockof any length; (7) U: forALockof unknown type; (8) x: forAn SCO openserver XenixLockOn part of the file; (9) X: forAn SCO openserver XenixLockon the entire file; Space:ifThere isNoLock.

• Type: File types, such as Dir, Reg, etc., common file types:

  (1) DIR: Table of Contents (2) CHR: denotes character type (3) BLK: Block device Type (4) Unix:unix domain socket (5) FIFO: FIFO (6) IPv4: Internet Protocol (IP) socket

• DEVICE: Specifies the name of the disk

• Size: Sizes of files

• Node: Index node (the identity of the file on disk)

• Name: Open the exact name of the file

Detailed usage Examples: http://linuxtools-rst.readthedocs.io/zh_CN/latest/tool/lsof.html

2. Troubleshooting

lsof | Grep-i Delete or lsof +d/tmp or lsof +d/tmp

Very many FPM processes are operating in the/tmp directory, after restarting FPM. Space release:

Some information:

Https://mp.weixin.qq.com/s/bEhuc1PPGPuYuA2mQArHLQ

Http://linuxtools-rst.readthedocs.io/zh_CN/latest/tool/lsof.html

[LINUX]DF disk 100%used alarm, du display directory status good troubleshooting