Memory troubleshooting methods and Related Files

5. Memory troubleshooting methods and related files [highlights]

For memory troubleshooting or OOM troubleshooting, the following files are generally involved. The analysis and settings of the following files are described below, which is also the focus of this document, check the memory information based on the file information. In fact, there may not be memory leaks, or there may be problems with some strategies, such as the increase in the number of threads, the application of buffer, and the intersection of release time.

5.1/proc/sys/Vm/min_free_kbytes

Min_free_kbytes is used to determine the threshold value for the system to recycle memory and control the idle memory of the system. The higher the value, the earlier the kernel starts to recycle memory, and the higher the idle memory.

You can use echo 65535>/proc/sys/Vm/min_free_kbytes to set the system reserved memory to 6 MB.

5.2/proc/sys/Vm/drop_caches

Clear system cache:

# Echo 1>/proc/sys/Vm/drop_caches clear page Cache

# Echo 2>/proc/sys/Vm/drop_caches clear File Cache

# Echo 3>/proc/sys/Vm/drop_caches clear cache and File Cache

5.3/proc/sysrq-trigger

When a sysrq command is triggered, the kernel prints information to the kernel's ring buffer and outputs it to the system console. This information is usually output to/var/log/messages through Syslog.

Sometimes, the system may be unable to respond, and syslogd may not be able to record this information. In this case, we recommend that you configure a serial port terminal to collect this information.

# Echo m>/proc/sysrq-trigger export memory allocation information

# Echo T>/proc/sysrq-trigger

# Echo P>/proc/sysrq-trigger: export the current CPU register information and Flag Information

# Echo W>/proc/sysrq-trigger: Dump information of tasks in the uninterrupted status

5.3.1 parsing echo m>/proc/sysrq-trigger output

 

[Email protected]:/home/Linux # echo m>/proc/sysrq-trigger

[Email protected]:/home/Linux # dmesg-C

Sysrq: show memory

Mem-Info:

DMA per-CPU:

CPU 0: Hi: 0, btch: 1 USD: 0

CPU 1: Hi: 0, btch: 1 USD: 0

Normal per-CPU:

CPU 0: Hi: 186, btch: 31 USD: 93

CPU 1: Hi: 186, btch: 31 USD: 140

Highmem per-CPU:

CPU 0: Hi: 186, btch: 31 USD: 185

CPU 1: Hi: 186, btch: 31 USD: 13

Active_anon: 22129 inactive_anon: 645isolated_anon: 0

Active_file: 52387 inactive_file: 81072isolated_file: 0

Unevictable: 0 dirty: 6 writeback: 0 unstable: 0

Free: 336695 slab_reclaimable: 12052slab_unreclaimable: 2825

Mapped: 11208 shmem: 892 pagetables: 795 bounce: 0

Free_cma: 0

DMA free: 15900kb min: 788kb low: 984 kbhigh: 1180kb active_anon: 0kb inactive_anon: 0kb active_file: 0kb inactive_file: 0 kb isolated (Anon): 0kb isolated (File ): 0kb present: 15992 kbmanaged: 1_mlocked: 0kb dirty: 0kb writeback: 0kb mapped: 0kb shmem: small: 0kb Bytes: 16kb kernel_stack: 0kb pagetables: 0 kbunstable: 0kb bounce: 0kb free_cma: 0kb writeback_tmp: 0kb pages_scanned: 0all_unreclaimable? No

Lowmem_reserve []: 0 843 1999 1999

Normal free: Minimum min: 42908 kblow: 53632kb high: 64360kb active_anon: 0kb inactive_anon: 0kb active_file: Maximum: 8520.kb unvictable: 0kb isolated (Anon): 0kb isolated (File ): 0 kbpresent: 897016kb managed: Lost mlocked: 0kb dirty: 24kb writeback: 0 kbmapped: 4kb shmem: 0kb memory: 48208kb memory: 1696kb pagetables: 0kb unstable: 0kb bounce: 0kb free_cma: 0 kbwri Teback_tmp: 0kb pages_scanned: 0 all_unreclaimable? No

Lowmem_reserve []: 0 0 9247 9247

Highmem free: 694940kb min: 512 kblow: large high: 29908kb active_anon: 88516kb failed: large: 133940kb inactive_file: 239184kb unevictable: 0kb isolated (Anon): 0 kbisolated (File ): 0kb present: Unsupported managed: Required mlocked: 0kb dirty: 0 kbwriteback: 0kb mapped: Required shmem: 3568kb rows: 0kb kernel_stack: 0kb pagetables: 3180kb unstable: 0 rows: 0kb free_cma : 0kb writeback_tmp: 0kb pages_scanned: 0 all_unreclaimable? No

Lowmem_reserve []: 0 0 0 0

DMA: 1 * 4kb (u) 1 * 8kb (u) 1 * 16kb (u) 0 * 32kb 2 * 64kb (u) 1 * 128kb (u) 1 * 256kb (u) 0 * 512kb 1 * 1024kb (u) 1 * 2048kb (r) 3 * 4096kb (m) = 15900kb

Normal: 1 * 4kb (m) 6 * 8kb (UEM) 3 * 16kb (UM) 2 * 32kb (EM) 2 * 64kb (UE) 2 * 128kb (UE) 2 * 256kb (UM) 2 * 512kb (UE) 1 * 1024kb (m) 1 * 2048kb (u) 154 * 4096kb (MR) = 635940kb

Highmem: 59 * 4kb (UM) 10 * 8kb (UM) 2 * 16kb (m) 58 * 32kb (UM) 48 * 64kb (UM) 4 * 128kb (UM) 2 * 256kb (UM) 1 * 512kb (m) 0*1024kb0 * 2048kb 168 * 4096kb (UMR) = 694940kb

Node 0 hugepages_total = 0hugepages_free = 0 hugepages_surp = 0 hugepages_size = 2048kb

134350 total pagecache pages

0 pages in SWAp Cache

Swap cache stats: Add 0, delete 0, find0/0

Free swap = 1045500kb

Total swap = 1045500kb

524287 pages Ram

295938 pages highmem

8303 pages reserved

647658 pages shared

93503 pages non-shared

[Email protected]:/home/Linux #

Pay attention to the following parameters:

Active_anon: anonymous ing page. These mappings are file-independent.

Active_file: Page mapped to a file. These mappings are related to files.

Mapped: size of device and file ing

Slab_reclaimable: the size of the kernel data structure cache, which can reduce the consumption caused by memory application and release.

Active_anon + active_file + mapped = application size

Active_anon + active_file + mapped + slab_reclaimable = all memory sizes currently used

In addition, we will pay attention to the following issues:

Normal: 1 * 4kb (m) 6 * 8kb (UEM) 3 * 16kb (UM) 2 * 32kb (EM) 2 * 64kb (UE) 2 * 128kb (UE) 2 * 256kb (UM) 2 * 512kb (UE) 1 * 1024kb (m) 1 * 2048kb (u) 154 * 4096kb (MR) = 635940kb

This is the memory information of Buddy System (partner system). If there is not much memory here, it is easy to encounter memory application failure or triggering OOM problems.

5.4 CAT/proc/slabinfo

In general, requests for smaller than one page of memory in the kernel program are completed through the kmalloc interface provided by the slab distributor. Because the cache allocated by slab is relatively small, we generally do not pay too much attention to it when analyzing the memory performance, because it will not affect the overall memory performance of the system, in addition, there are fewer problems with kmalloc in the kernel and more problems with the application layer.

However, you can use the following information to obtain the slab Cache Usage:

[Email protected]:/home/Linux # Cat/proc/slabinfo

View the slab information. In addition to viewing the slabinfo file, you can also run the/usr/bin/slabtop command:

The above is the information of a/proc/slabinfo file.

[Email protected]:/home/Linux #/usr/bin/slabtop

Active/total objects (% used): 311009/312759 (99.4%)

Active/total slabs (% used): 7308/7308 (100.0%)

Active/total caches (% used): 66/100 (66.0%)

Active/total size (% used): 58479.13 K/58915.22 K (99.3%)

Minimum/average/maximum object: 0.01 K/0.19 K/8.00 K

 

Objs active use OBJ size slabs OBJ/slab cache size name

73949 73949 100% 0.05 K 1013 73 4052 K buffer_head

66976 66686 99% 0.12 K 2093 32 8372 K dentry

47100 47100 100% 0.63 K 1884 25 30144 K ext4_inode_cache

Meaning of the output field of this command:

Objs-the total number of objects (memory blocks), including those in use (allocated), and some spares not inuse.

Active-the number of objects (memoryblocks) that are in use (allocated ).

Use-Percentage of total objects thatare active. (active/objs) (100 ))

OBJ size-the size of the objects.

Slabs-the total number of slabs.

OBJ/slab-the number of objects thatfit into a slab.

Cache size-the cache size of theslab.

Name-the name of the slab.

5.5 CAT/proc/meminfo5.5.1 important information

For this file, active, active (Anon), active (file), mapped, and slab are important. Similar to sysrq-trigger, the following computation relationship exists:

Active = active (Anon) + active (file)

Active (Anon) + AC tive (File) + mapped = Application

Active (Anon) + AC tive (File) + mapped + slab = total memory used by the System

5.5.2 description of fields in the meminfo File

$ CAT/proc/meminfo

Memtotal: 2052440 kb // total memory

Memfree: 50004 kb // idle memory

Buffers: 19976 kb // buffer size for the object

Cached: 436412 kb // the size of the cache memory (http://baike.baidu.com/view/496990.htm) used

Swapcached: 19864 kb // swap space used for high-speed buffer storage

Active: 1144512 kb // The page file size of the cache in active use

Inactive: 732788 kb // the size of the cache page file that is not frequently used

Active (Anon): 987640 kb // Anon: soon

Inactive (Anon): 572512 KB

Active (File): 156872 KB

Inactive (File): 160276 KB

Unevictable: 8 KB

Mlocked: 8 KB

Hightotal: 1177160 kb // The total and free amountof memory, in kilobytes, that is not directly mapped into kernel space.

Highfree: 7396 kb // The hightotal value canvary based on the type of Kernel used.

Lowtotal: 875280 kb // The total and free amountof memory, in kilobytes, that is directly mapped into kernel space. used.

Lowfree: 42608 kb // The lowtotal value canvary based on the type of Kernel

Swaptotal: 489940 kb // the total size of swap space

Swapfree: 450328 kb // free swap space

Dirty: 104 KB // size of the disk to be written back

Writeback: 0 kb // size of the data being written back

Anonpages: 1408256 kb // unmapped page size

Mapped: 131964 kb // device and file ing size

Slab: 37368 kb // the size of the kernel data structure cache, which can reduce the consumption caused by memory application and release.

Sreclaimable: 14164 kb // recoverable slab size

Sunreclaim: 23204 kb // size of unrecoverable Slab: 23204 + 14164 = 37368

Pagetables: 13308 kb // the size of the index table for managing memory pages

Nfs_unstable: 0 kb // the size of the unstable page table

Bounce: 0 kb // bounce: returned

Writebacktmp: 0 kb //

Commitlimit: 1516160 KB

Committed_as: 2511900 KB

Vmalloctotal: 122880 kb // virtual memory size

Vmallocused: 28688 kb // The used virtual memory size

Vmallocchunk: 92204 KB

Hugepages_total: 0 // large page allocation

Hugepages_free: 0

Hugepages_rsvd: 0

Hugepages_surp: 0

Hugepagesize: 2048 KB

Directmap4 KB: 10232 KB

Directmap2m: 899072 KB

5.6 top info

Virt: virtual memory usage virtual memory
1. The virtual memory size required by the process, including the database, code, and data used by the Process
2. If the process requests 100 MB of memory but only uses 10 MB, it will increase by MB instead of the actual usage.

Res: resident memory usage resident memory (physical memory)
1. Memory size currently used by the process, excluding swap out
2. Sharing of other processes
3. If you apply for MB of memory, the actual usage of 10 MB will only increase by 10 MB, which is opposite to virt.
4. When the Library occupies memory, it only counts the memory size occupied by the loaded library files.

SHR: shared memory
1. In addition to the shared memory of processes, it also includes the shared memory of other processes.
2. Although the process only uses several shared library functions, it contains the size of the entire shared library.
3. Calculate the physical memory occupied by a process. formula: res-SHR
4. After swap out, it will be downgraded

Data
1. Memory occupied by data. If top is not displayed, press F to display it.
2. The data space required by the program is actually used during running.

You can use top's Internal commands to control the display mode of processes. The internal command is as follows:
S-change the screen update frequency
L-Disable or enable the first line of top information in the first part.
T-Disable or enable the expression of the first part of the second line of tasks and the third line of CPUs Information
M-Disable or enable the expression of the first part of the fourth line of MEM and the fifth line of swap information
N-process list in the order of PID size
P-process list in the order of CPU usage
M-sort the process list in the order of memory usage
H-Show Help
N-set the number of processes displayed in the Process List
Q-exit top
S-change the screen update cycle

Serial Number Column name meaning
A pid process ID
B ppid parent process ID
C ruser real User Name
D uid: User ID of the process owner
E User: User Name of the process owner
Group name of the owner of the F group process
The terminal name of the G tty startup process. Processes not started from the terminal are displayed?
H PR priority
I ni nice value. A negative value indicates a high priority, and a positive value indicates a low priority.
The final CPU used by j p is only meaningful in multiple CPU Environments
K % percentage of CPU time used since the last CPU update to the present
L total CPU time used by the time process, in seconds
M Time + the total CPU time used by the process, in the unit of 1/100 seconds
N % mem percentage of physical memory used by the Process
O total virtual memory used by the virt process, in KB. Virt = swap + Res
The swap size in the virtual memory used by the P SWAp process, in KB.
The size of the physical memory used by the Q res process, Not swapped out, in KB. Res = code + Data
The physical memory occupied by the R code executable code, in KB.
Physical memory occupied by other parts (Data Segment + stack) except the executable code of S data, in KB
T SHR shared memory size, in KB
U nflt page error count
The number of modified pages that the last write to the present by V.
W s Process status. (D = non-disruptive sleep state, r = running, S = sleep, t = tracking/stopping, Z = zombie process)
X command name/command line
Y wchan: if the process is sleep, the system function name in sleep is displayed.
Z flags task flag, refer to sched. h

By default, only important PID, user, PR, Ni, virt, res, SHR, S, % CPU, % MEM, time +, and command columns are displayed. You can use the shortcut keys below to change the display content.

You can use the f key to select the displayed content. Press F to display the column list. Press A-Z to display or hide the corresponding column, and press enter to confirm.
Press the O key to change the Column Display sequence. A lower-case a-Z can move the corresponding column to the right, while an upper-case A-Z can move the corresponding column to the left. Press enter.
Press the f or O key in upper case, and then press a-Z to sort the process according to the corresponding column. The uppercase r key can reverse the current sorting.

5.7 free command

 

There are four rows of free output, and the fourth is the information in the SWAp area, which is the total amount of exchange, the amount of used and the number of idle swap areas)

The second and third rows of free output locations indicate memory usage. The first column is total, the second column is used, and the third column is free ).

The output in line 3 is determined by the operating system (OS. In other words, from the OS perspective, there are a total of computers:

24677460kb (unit: KB) physical memory;

23276064kb is used in these physical memory;

1401396kb is also available;

Row 3:

-Buffers/cache indicates the amount of memory that an application deems the system to be used;

+ Buffers/cache indicates the amount of memory that an application thinks the system has;

Other files in the 5.8/proc/sys/Vm/directory

1./proc/sys/Vm/block_dump
This file indicates whether the block debug mode is enabled. It is used to record all read/write and dirtyblock write-back actions.
Default setting: 0. Disable blockdebug mode.

2./proc/sys/Vm/dirty_background_ratio
This file indicates the percentage of dirty data in the system's overall memory. At this time, the pdflush process is triggered to write the dirty data back to the disk.
Default setting: 10

3./proc/sys/Vm/dirty_expire_centisecs
This file indicates that if the residence time of dirty data exceeds this value in the memory, the pdflush process will write the data back to the disk next time.
Default setting: 3000 (1/100 seconds)

4./proc/sys/Vm/dirty_ratio
This file indicates the percentage of dirty data generated by the process to the overall memory of the system. At this time, the process writes the dirty data back to the disk.
Default setting: 40

5./proc/sys/Vm/dirty_writeback_centisecs
This file indicates how long the pdflush process writes dirty data back to the disk at a periodic interval.
Default setting: 500 (1/100 seconds)

6./proc/sys/Vm/vfs_cache_pressure
This file indicates that the kernel recycles the preference for directory and inode cache memory. The default value 100 indicates that the kernel will keep directory and inode cache in a reasonable percentage based on pagecache and swapcache; if this value is lower than 100, the kernel tends to retain directory and inode cache. If this value is increased to more than 100, the kernel tends to recycle directory and inode cache.
Default: 100

7./proc/sys/Vm/min_free_kbytes
This file indicates the minimum number of idle memory (Kbytes) reserved by the Linux VM ).
Default setting: 724 (MB physical memory)

8./proc/sys/Vm/nr_pdflush_threads
This file indicates the number of currently running pdflush processes. When the I/O load is high, the kernel will automatically add more pdflush processes.
Default setting: 2 (read-only)

9./proc/sys/Vm/overcommit_memory
This file specifies the kernel memory allocation policy. The value can be 0, 1, or 2.
0 indicates that the kernel will check whether there is enough available memory for use by the process. If there is enough available memory, the memory application will be allowed; otherwise, the memory application will fail, and return the error to the application process.
1 indicates that the kernel allows all physical memory allocation regardless of the current memory status.
2, indicating that the kernel can allocate more memory than the total physical memory and swap space (refer to overcommit_ratio ).
Default Value: 0.

10./proc/sys/Vm/overcommit_ratio
This file indicates that if overcommit_memory = 2, the percentage of memory can be overloaded. The following formula is used to calculate the overall available memory of the system.
System allocable memory = swap space + physical memory * overcommit_ratio/100
Default setting: 50 (%)

11./proc/sys/Vm/page-Cluster
This file indicates the number of pages written when writing to the swap area once. 0 indicates 1 page, 1 indicates 2 pages, and 2 indicates 4 pages.
Default settings: 3 (3 to the power of 2, 8 pages)

12./proc/sys/Vm/swapiness
This file indicates the extent to which the system performs swap. The higher the value (0-100), the more likely disk swap will occur.

 

Memory troubleshooting methods and Related Files