Memory troubleshooting tools and related documents introduction

[-]

1. Memory troubleshooting tools and related documents introduction focus
1. 1 procsysvmmin_free_kbytes
2. 2 procsysvmdrop_caches
3. 3 Procsysrq-trigger
1. 31 interpretation of the echo m Procsysrq-trigger output
4. 4 Cat Procslabinfo
5. 5 Cat Procmeminfo
1. 51 Important Information
6. 6 Top Information
7. 7 Free Command
8. 8 other files in the PROCSYSVM directory

5. Memory troubleshooting tools and related Documents introduction "Focus"

For memory troubleshooting, or oom troubleshooting, will generally involve the following files, the following file analysis and settings described below, this is the focus of this document, the following troubleshooting memory information or to be based on these file information to troubleshoot. In fact, there is not necessarily a memory leak, it may be some strategy problems, such as the increase in the number of threads, buffer application, release time intersection and so on.

5.1/proc/sys/vm/min_free_kbytes

The min_free_kbytes is used to determine the threshold at which the system starts to reclaim memory and to control the system's free memory. The higher the value, the sooner the kernel starts to reclaim memory, and the higher the free memory.

You can use the following methods: Echo 65535 >/proc/sys/vm/min_free_kbytes Sets the system reserved memory to 6M.

5.2/proc/sys/vm/drop_caches

Clear System cache:

#echo 1 >/proc/sys/vm/drop_caches Clean page cache

#echo 2 >/proc/sys/vm/drop_caches clean up file cache

#echo 3 >/proc/sys/vm/drop_caches cleanup also cache and file cache

5.3/proc/sysrq-trigger

When a SYSRQ command is triggered, the kernel will print the message to the kernel's ring buffer and output it to the system console. This information is also typically output to/var/log/messages via a syslog.

Sometimes, the system may not be responding, and syslogd may not be able to log this information. In this case, it is recommended that you configure a serial terminal to collect this information.

#echo m >/proc/sysrq-trigger Exporting memory allocation information

#echo T >/proc/sysrq-trigger Exporting Thread state information

#echo p >/proc/sysrq-trigger to export current CPU register information and flag bit information

#echo w >/proc/sysrq-trigger will go into the uninterrupted state of the mission information dump out

5.3.1 Analysis of echo m >/proc/sysrq-trigger output

[Email protected]:/home/linux# echo M >/proc/sysrq-trigger

[Email protected]:/home/linux# dmesg-c

Sysrq:show Memory

Mem-info:

DMA PER-CPU:

CPU 0:hi:0, Btch:1 usd:0

CPU 1:hi:0, Btch:1 usd:0

Normal PER-CPU:

CPU 0:hi:186, btch:31 usd:93

CPU 1:hi:186, btch:31 usd:140

Highmem PER-CPU:

CPU 0:hi:186, btch:31 usd:185

CPU 1:hi:186, btch:31 usd:13

active_anon:22129 inactive_anon:645isolated_anon:0

active_file:52387 inactive_file:81072isolated_file:0

unevictable:0 Dirty:6 writeback:0unstable:0

free:336695 slab_reclaimable:12052slab_unreclaimable:2825

mapped:11208 shmem:892 pagetables:795bounce:0

free_cma:0

DMA free:15900kb min:788kb low:984kbhigh:1180kb active_anon:0kb inactive_anon:0kb active_file:0kb inactive_file:0 KBUNEVICTABLE:0KB Isolated (anon): 0kB isolated (file): 0kB present:15992kbmanaged:15916kb mlocked:0kb dirty:0kb WRITEBACK:0KB mapped:0kb shmem:0kbslab_reclaimable:0kb slab_unreclaimable:16kb Kernel_stack:0kB pagetables:0 KBUNSTABLE:0KB bounce:0kb free_cma:0kb writeback_tmp:0kb pages_scanned:0all_unreclaimable? No

Lowmem_reserve[]: 0 843 1999 1999

Normal free:635940kb min:42908kblow:53632kb high:64360kb active_anon:0kb inactive_anon:0kb active_file : 75608kbinactive_file:85104kb unevictable:0kb Isolated (anon): 0kB isolated (file): 0kbpresent:897016kb managed : 864652kB mlocked:0kb dirty:24kb writeback:0kbmapped:4kb shmem:0kb slab_reclaimable:48208kb slab_unreclaimable : 11284kbkernel_stack:1696kb pagetables:0kb unstable:0kb bounce:0kb free_cma:0kbwriteback_tmp:0kb pages_scanned:0 All _unreclaimable? No

Lowmem_reserve[]: 0 0 9247 9247

Highmem free:694940kb min:512kblow:15208kb high:29908kb active_anon:88516kb inactive_anon:2580kbactive_file : 133940kB inactive_file:239184kb unevictable:0kb Isolated (anon): 0kBisolated (file): 0kB PRESENT:1183624KB Managed : 1183624kB mlocked:0kb dirty:0kbwriteback:0kb mapped:44828kb shmem:3568kb slab_reclaimable:0kbslab_unreclaimable:0 KB kernel_stack:0kb pagetables:3180kb unstable:0kbbounce:0kb free_cma:0kb writeback_tmp:0kb pages_scanned:0 All_ Unreclaimable? No

Lowmem_reserve[]: 0 0 0 0

DMA:1*4KB (U) 1*8kb (u) 1*16kb (u) 0*32kb 2*64kb (U) 1*128kb (u) 1*256kb (u) 0*512kb 1*1024kb (U) 1*2048kb (R) 3*4096kb (M) = 15900kB

NORMAL:1*4KB (M) 6*8kb (UEM) 3*16kb (UM) 2*32kb (EM) 2*64KB (UE) 2*128KB (UE) 2*256kb (UM) 2*512KB (UE) 1*1024KB (M) 1*2048 KB (U) 154*4096kb (MR) = 635940kB

HIGHMEM:59*4KB (UM) 10*8kb (UM) 2*16kb (M) 58*32kb (UM) 48*64kb (UM) 4*128kb (UM) 2*256kb (UM) 1*512kb (m) 0*1024kb0*2048k B 168*4096kb (UMR) = 694940kB

Node 0 hugepages_total=0hugepages_free=0 Hugepages_surp=0 hugepages_size=2048kb

134350 Total Pagecache Pages

0 pages in swap cache

Swap Cache Stats:add 0, delete 0, find0/0

Free swap = 1045500kB

Total Swap = 1045500kB

524287 pages RAM

295938 pages Highmem

8303 pages reserved

647658 pages Shared

93503 pages non-shared

[Email protected]:/home/linux#

The main concerns are the following parameters:

Active_anon: Anonymously mapped pages, which are file-independent

Active_file: File-mapped pages, these mappings are related to files

Mapped: size of device and file mappings

Slab_reclaimable: The size of the kernel data structure cache to reduce the consumption of application and release memory

Active_anon + active_file + mapped = size used by the app

Active_anon + active_file + Mapped +slab_reclaimable = All memory sizes currently in use

In addition, we will also generally focus on:

NORMAL:1*4KB (M) 6*8kb (UEM) 3*16kb (UM) 2*32kb (EM) 2*64KB (UE) 2*128KB (UE) 2*256kb (UM) 2*512KB (UE) 1*1024KB (M) 1*2048 KB (U) 154*4096kb (MR) = 635940kB

This is the memory information of the buddy system (partner systems), and if there is not much memory here, it is also very easy to fail the memory request or to trigger the oom problem

5.4 Cat/proc/slabinfo

In general, requests for small chunks of memory smaller than one page in kernel programs are done through the interface Kmalloc provided by the slab allocator. Because slab allocated cache is relatively small cache, so in general we analyze memory performance, there is not too much concern here, because of its overall memory performance of the system is not too large, and the kernel of the kmalloc relatively less problems, the application layer of more problems.

However, you can use the following information to obtain the usage of the slab cache:

[Email protected]:/home/linux# cat/proc/slabinfo

To view slab information, you can also execute the/usr/bin/slabtop command in addition to the Slabinfo file:

The above is a/proc/slabinfo file information

[Email Protected]:/home/linux#/usr/bin/slabtop

Active/total Objects (% used): 311009/312759 (99.4%)

Active/total Slabs (% used): 7308/7308 (100%)

Active/total Caches (% used): 66/100 (66%)

Active/total Size (% used): 58479.13k/58915.22k (99.3%)

Minimum/average/maximum object:0.01k/0.19k/8.00k

OBJS ACTIVE use OBJ size slabs obj/slab CACHE size NAME

73949 73949 100% 0.05K 1013 4052K Buffer_head

66976 66686 99% 0.12K 2093 8372K dentry

47100 47100 100% 0.63K 1884 30144K Ext4_inode_cache

The meaning of the output field of the command:

Objs-the Total number of objects (memory blocks), including those on use (allocated), and some spares not inuse.

Active-the Number of objects (Memoryblocks) that is in use (allocated).

Use-percentage of total objects thatare active. ((ACTIVE/OBJS) (100))

OBJ size-the SIZE of the objects.

Slabs-the total number of slabs.

Obj/slab-the number of objects Thatfit into a SLAB.

Cache size-the Cache SIZE of Theslab.

Name-the name of the slab.

5.5 cat/proc/meminfo5.5.1 Important Information

For this file, it is important that active, active (anon), active (file), Mapped, Slab, similar to the one described in Sysrq-trigger, have the following computational relationships:

Active = Active (anon) + active (file)

Active (anon) +ac tive (file) +mapped = App

Active (anon) +ac tive (file) +mapped +slab= system uses a total of memory

5.5.2 meminfo file each field description

$cat/proc/meminfo

memtotal:2052440 KB//Total Memory

memfree:50004 KB//Free memory

buffers:19976 KB//buffer size to file

cached:436412 KB//Buffer memory (http://baike.baidu.com/view/496990.htm) size used

swapcached:19864 KB//Swap space size for cache storage

active:1144512 KB//cache memory paging file size in active use

inactive:732788 KB//Infrequently used buffer memory paging file size

Active (anon): 987640 KB//anon: Soon

Inactive (anon): 572512 KB

Active (file): 156872 KB

Inactive (file): 160276 KB

Unevictable:8 KB

Mlocked:8 KB

hightotal:1177160 KB//the Total and free amountof memory, in kilobytes, that's not directly mapped to kernel SP Ace.

highfree:7396 KB//The Hightotal value canvary based on the type of kernel used.

lowtotal:875280 KB//The total and free amountof memory, in kilobytes, that's directly mapped into kernel space  . Used.

lowfree:42608 KB//the lowtotal value canvary based on the type of kernel

swaptotal:489940 KB//Swap space Total size

swapfree:450328 KB//Free swap space

dirty:104 KB//Waits to be written back to the size of the disk

writeback:0 KB//size that is being written back

anonpages:1408256 KB//size of unmapped pages

mapped:131964 KB//size of device and file mappings

slab:37368 KB//Kernel data structure cache size to reduce the cost of requesting and freeing memory

sreclaimable:14164 KB//recoverable slab size

sunreclaim:23204 KB//non-recoverable size of slab 23204+14164=37368

pagetables:13308 KB//Managing the size of index tables for memory paging

nfs_unstable:0 KB//size of unstable page table

bounce:0 KB//bounce: Return

writebacktmp:0 KB//

commitlimit:1516160 KB

committed_as:2511900 KB

vmalloctotal:122880 KB//virtual memory size

vmallocused:28688 KB//virtual memory size already in use

vmallocchunk:92204 KB

hugepages_total:0//Allocation of large pages

hugepages_free:0

hugepages_rsvd:0

hugepages_surp:0

hugepagesize:2048 KB

directmap4k:10232 KB

directmap2m:899072 KB

5.6 Top Information

Virt:virtual Memory usage Virtual RAM
1. The process "required" virtual memory size, including the library, code, data, etc. used by the process
2, if the process to apply for 100m of memory, but the actual use of only 10m, then it will grow 100m, rather than the actual use of the volume

Res:resident Memory usage resident RAM (physical memory)
1. The memory size currently used by the process, but does not include swap out
2. Sharing with other processes
3, if the application of 100m of memory, the actual use of 10m, it only grows 10m, and virt opposite
4, about the memory consumption of the library, it only statistics the size of the loaded library file memory

Shr:shared Memory Shared
1, in addition to the shared memory of the process itself, also includes the shared memory of other processes
2, although the process uses only a few shared library functions, but it contains the entire size of the shared library
3. Calculate the physical memory size formula that a process occupies: res–shr
4, swap out, it will be lowered

DATA
1, the memory occupied by the data. If top is not displayed, press the F key to display it.
2, the real data space required by the program is really in operation to use.

In the top run, you can control how processes are displayed through the internal commands of top. The internal commands are as follows:
S-Change the screen update frequency
L – Turn off or turn on the first section of the first line top information representation
T-Close or turn on the first part of the second line the CPUs information representation of the Tasks and the third row
M – Turn off or turn on the first part of the four-line Mem and fifth line swap information representation
n – the list of processes in the order of the PID size
P – List of processes in the order of CPU utilization
M – List of processes in the order of memory utilization
H – Display Help
n – Set the number of processes that are displayed in the process list
Q – Exit Top
S-Change the screen update cycle

Ordinal column name meaning
A PID process ID
b PPID Parent Process ID
C ruser Real User name
D UID Process Owner's user ID
e username of user process Owner
Group name of the F group Process owner
The terminal name of the G TTY boot process. Processes that are not started from the terminal are displayed as?
H PR-Priority
I NI nice value. Negative values indicate high priority, positive values indicate low priority
The last CPU used by J P is meaningful only in a multi-CPU environment
K%cpu CPU time consumption percentage last updated to current
The total CPU time, in seconds, used by the timing process
The total CPU time used by the M time+ process, in units 1/100 seconds
n the percentage of physical memory used by the%MEM process
o The total amount of virtual memory used by the VIRT process, in kilobytes. Virt=swap+res
The P swap process uses the size of the virtual memory, which is swapped out, in kilobytes.
The size of the physical memory, in kilobytes, that the Q RES process uses and has not been swapped out. Res=code+data
R code executable code occupies the physical memory size, in kilobytes
The amount of physical memory that is used outside of the S data executable code (data segment + stack), in kilobytes
T SHR shared memory size, in kilobytes
U Nflt page Error count
V NDRT the number of pages that have been modified in the last write to now.
W S Process state. (d= non-disruptive sleep status, r= run, s= sleep, t= track/Stop, z= zombie process)
x command name/command line
Y Wchan If the process is sleeping, the system function name in sleep is displayed
Z Flags task Flag, reference sched.h

By default, only the more important PID, USER, PR, NI, VIRT, RES, SHR, S,%cpu,%MEM, time+, and COMMAND columns are displayed. You can change the display by using the following shortcut keys.

The F key allows you to select what is displayed. Press the F key to display a list of columns, press A-Z to show or hide the corresponding column, and then press ENTER to confirm.
Press the O key to change the order in which the columns are displayed. A-Z in the lower case moves the corresponding column to the right, while the uppercase A-Z moves the corresponding column to the left. Finally, press ENTER to confirm.
Press the uppercase F or O key, and then press A-Z to sort the process by the appropriate column. The uppercase R key can reverse the current sort.

5.7 Free Command

The output of free has a total of four lines, the fourth behavior swap information, which is the total amount of the exchange, the amount of usage (used), and the number of idle swap areas

The second and third rows of the free output indicate memory usage. The first column is total, the second column is the usage (used), and the third column is the available amount (free).

The output from line 2nd is viewed from the operating system (OS). That is, from the OS point of view, the computer has a total of:

24677460KB (the default is KB for free) physical memory;

23276064KB was used in these physical memory;

Also use 1401396KB is available;

Line 3rd:

-buffers/cache, which indicates how much memory an application thinks the system is using;

+buffers/cache, which indicates how much memory an application thinks the system has;

Other files in the 5.8/proc/sys/vm/directory

1./proc/sys/vm/block_dump
This file indicates whether to open block debug mode, which is used to record all read-write and Dirtyblock write-back actions.
Default setting: 0, disable Blockdebug mode

2./proc/sys/vm/dirty_background_ratio
This file represents the percentage of dirty data that reaches the overall system memory, triggering the pdflush process to write dirty data back to disk.
Default setting: 10

3./proc/sys/vm/dirty_expire_centisecs
The file indicates that if the dirty data resides in memory longer than this value, the Pdflush process will write the data back to disk the next time.
Default setting: 3000 (1/100 sec)

4./proc/sys/vm/dirty_ratio
This file indicates that if the dirty data generated by the process reaches the percentage of the system's overall memory, the process itself writes the dirty data back to disk.
Default setting: 40

5./proc/sys/vm/dirty_writeback_centisecs
This file indicates how often the Pdflush process writes dirty data back to disk.
Default setting: 500 (1/100 sec)

6./proc/sys/vm/vfs_cache_pressure
This file indicates that the kernel recycles the memory used by the directory and Inode caches, and the default value of 100 means that the kernel will keep the directory and inode caches at a reasonable percentage based on Pagecache and Swapcache. Lowering this value below 100 causes the kernel to tend to retain the directory and Inode caches, and increasing this value by more than 100 will cause the kernel to tend to recycle the directory and Inode caches
Default setting: 100

7./proc/sys/vm/min_free_kbytes
This file represents the minimum amount of free memory (Kbytes) that the Linux VM is forced to keep.
Default setting: 724 (512M physical memory)

8./proc/sys/vm/nr_pdflush_threads
This file represents the number of Pdflush processes currently running, and the kernel will automatically add more Pdflush processes with high I/O load.
Default setting: 2 (Read only)

9./proc/sys/vm/overcommit_memory
This file specifies the kernel's policy for memory allocation, which can be 0, 1, 2.
0, indicates that the kernel will check for sufficient available memory to be used by the process, and if sufficient memory is available, the memory request is allowed; otherwise, the memory request fails and the error is returned to the application process.
1, which means that the kernel allows all physical memory to be allocated regardless of the current memory state.
2, which indicates that the kernel allows allocating more memory than the sum of all physical memory and swap space (refer to Overcommit_ratio).
Default setting: 0

Ten./proc/sys/vm/overcommit_ratio
The file indicates that if overcommit_memory=2, the percentage of memory that can be overloaded, the overall available memory of the system is calculated by the following formula.
System assignable Memory = Swap space + physical memory *overcommit_ratio/100
Default setting: 50 (%)

/proc/sys/vm/page-cluster.
This file represents the number of pages written once to the swap area, 0 for 1 pages, 1 for 2 pages, and 2 for 4 pages.
Default setting: 3 (2 of 3 parties, 8 pages)

/proc/sys/vm/swapiness.
The file represents the degree to which the system is exchanging behavior, and the higher the value (0-100), the more likely the disk exchange will occur.

Memory troubleshooting tools and related documents introduction